2023-12-25T20:50:18.939 INFO:root:teuthology version: 0.0.1.dev226+g9680d2a 2023-12-25T20:50:18.940 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --name yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi --verbose --archive /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500977 -- /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500977/orig.config.yaml 2023-12-25T20:50:18.989 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-25T20:50:19.090 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500977 branch: wip-yuri5-testing-2023-12-15-0747-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7500977' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri5-testing-2023-12-15-0747-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri5-testing-2023-12-15-0747-pacific sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 2834 sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri5-testing-2023-12-15-0747-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 targets: smithi061.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNGWUdfyqeJ/EhV66Hng+Yju49pNNiEPvGvBkWe82OrfjGFxgf42CPBREXpFsKHIDf9ZSodY2pEzA9nyOE3h6KI= smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBXyRFTdwGQLNHrZkx91RjrtdgcYVey/TBmxRFQkadX/e83vU7kXA9bqPckj6BeBlGlKQxXF7BVv32LD9zoZ8ig= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9680d2aba1e76f64ade295c81599c97907457848 timestamp: 2023-12-25_17:28:36 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-12-25T20:50:19.091 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa; will attempt to use it 2023-12-25T20:50:19.091 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks 2023-12-25T20:50:19.091 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-25T20:50:19.093 INFO:teuthology.task.internal:Checking packages... 2023-12-25T20:50:19.116 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2' 2023-12-25T20:50:19.116 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-25T20:50:19.117 INFO:teuthology.packaging:ref: None 2023-12-25T20:50:19.117 INFO:teuthology.packaging:tag: None 2023-12-25T20:50:19.117 INFO:teuthology.packaging:branch: wip-yuri5-testing-2023-12-15-0747-pacific 2023-12-25T20:50:19.117 INFO:teuthology.packaging:sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T20:50:19.117 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri5-testing-2023-12-15-0747-pacific 2023-12-25T20:50:19.339 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-385.gb14a0a10 2023-12-25T20:50:19.340 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-25T20:50:19.347 INFO:teuthology.task.internal:no buildpackages task found 2023-12-25T20:50:19.347 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-25T20:50:19.354 INFO:teuthology.task.internal:Saving configuration 2023-12-25T20:50:19.375 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-25T20:50:19.381 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-25T20:50:19.403 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi061.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500977', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-25 20:45:16.868182', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNGWUdfyqeJ/EhV66Hng+Yju49pNNiEPvGvBkWe82OrfjGFxgf42CPBREXpFsKHIDf9ZSodY2pEzA9nyOE3h6KI='} 2023-12-25T20:50:19.429 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500977', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-25 20:45:16.866965', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBXyRFTdwGQLNHrZkx91RjrtdgcYVey/TBmxRFQkadX/e83vU7kXA9bqPckj6BeBlGlKQxXF7BVv32LD9zoZ8ig='} 2023-12-25T20:50:19.429 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-25T20:50:19.435 INFO:teuthology.task.internal:roles: ubuntu@smithi061.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-12-25T20:50:19.435 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-12-25T20:50:19.436 INFO:teuthology.run_tasks:Running task console_log... 2023-12-25T20:50:19.507 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3e73605e50>, signals=[15]) 2023-12-25T20:50:19.508 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-25T20:50:19.514 INFO:teuthology.task.internal:Opening connections... 2023-12-25T20:50:19.514 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi061.front.sepia.ceph.com 2023-12-25T20:50:19.516 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T20:50:19.584 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2023-12-25T20:50:19.585 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T20:50:19.651 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-25T20:50:19.657 DEBUG:teuthology.orchestra.run.smithi061:> uname -m 2023-12-25T20:50:19.681 INFO:teuthology.orchestra.run.smithi061.stdout:x86_64 2023-12-25T20:50:19.682 DEBUG:teuthology.orchestra.run.smithi061:> cat /etc/os-release 2023-12-25T20:50:19.745 INFO:teuthology.orchestra.run.smithi061.stdout:NAME="CentOS Stream" 2023-12-25T20:50:19.746 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION="8" 2023-12-25T20:50:19.746 INFO:teuthology.orchestra.run.smithi061.stdout:ID="centos" 2023-12-25T20:50:19.746 INFO:teuthology.orchestra.run.smithi061.stdout:ID_LIKE="rhel fedora" 2023-12-25T20:50:19.746 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_ID="8" 2023-12-25T20:50:19.746 INFO:teuthology.orchestra.run.smithi061.stdout:PLATFORM_ID="platform:el8" 2023-12-25T20:50:19.746 INFO:teuthology.orchestra.run.smithi061.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-25T20:50:19.746 INFO:teuthology.orchestra.run.smithi061.stdout:ANSI_COLOR="0;31" 2023-12-25T20:50:19.746 INFO:teuthology.orchestra.run.smithi061.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-25T20:50:19.747 INFO:teuthology.orchestra.run.smithi061.stdout:HOME_URL="https://centos.org/" 2023-12-25T20:50:19.747 INFO:teuthology.orchestra.run.smithi061.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-25T20:50:19.747 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-25T20:50:19.747 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-25T20:50:19.748 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server 2023-12-25T20:50:19.769 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2023-12-25T20:50:19.791 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2023-12-25T20:50:19.792 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2023-12-25T20:50:19.853 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="CentOS Stream" 2023-12-25T20:50:19.853 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="8" 2023-12-25T20:50:19.853 INFO:teuthology.orchestra.run.smithi174.stdout:ID="centos" 2023-12-25T20:50:19.854 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="rhel fedora" 2023-12-25T20:50:19.854 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="8" 2023-12-25T20:50:19.854 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el8" 2023-12-25T20:50:19.854 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-25T20:50:19.854 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31" 2023-12-25T20:50:19.854 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-25T20:50:19.854 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://centos.org/" 2023-12-25T20:50:19.855 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-25T20:50:19.855 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-25T20:50:19.855 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-25T20:50:19.855 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2023-12-25T20:50:19.877 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-25T20:50:19.886 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-25T20:50:19.891 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-25T20:50:19.892 DEBUG:teuthology.orchestra.run.smithi061:> test '!' -e /home/ubuntu/cephtest 2023-12-25T20:50:19.895 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2023-12-25T20:50:19.915 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-25T20:50:19.920 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-25T20:50:19.921 DEBUG:teuthology.orchestra.run.smithi061:> test -z $(ls -A /var/lib/ceph) 2023-12-25T20:50:19.957 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2023-12-25T20:50:19.988 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-25T20:50:20.044 INFO:teuthology.run_tasks:Running task kernel... 2023-12-25T20:50:20.061 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-25T20:50:20.061 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-25T20:50:20.061 DEBUG:teuthology.orchestra.run.smithi061:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-25T20:50:20.062 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-25T20:50:20.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T20:50:20.082 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2023-12-25T20:50:20.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T20:50:20.083 DEBUG:teuthology.orchestra.run.smithi061:> uname -r 2023-12-25T20:50:20.143 INFO:teuthology.orchestra.run.smithi174.stdout:4.18.0-489.el8.x86_64 2023-12-25T20:50:20.144 INFO:teuthology.task.kernel:Running kernel on smithi174: 4.18.0-489.el8.x86_64 2023-12-25T20:50:20.144 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum install -y kernel 2023-12-25T20:50:20.146 INFO:teuthology.orchestra.run.smithi061.stdout:4.18.0-489.el8.x86_64 2023-12-25T20:50:20.146 INFO:teuthology.task.kernel:Running kernel on smithi061: 4.18.0-489.el8.x86_64 2023-12-25T20:50:20.146 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum install -y kernel 2023-12-25T20:50:25.161 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 8 - AppStream 14 MB/s | 35 MB 00:02 2023-12-25T20:50:25.181 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 8 - AppStream 14 MB/s | 35 MB 00:02 2023-12-25T20:50:50.767 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 8 - BaseOS 3.0 MB/s | 57 MB 00:19 2023-12-25T20:50:52.104 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 8 - BaseOS 2.8 MB/s | 57 MB 00:20 2023-12-25T20:50:59.075 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 8 - Extras 79 kB/s | 18 kB 00:00 2023-12-25T20:50:59.842 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 8 - Extras common packages 12 kB/s | 6.9 kB 00:00 2023-12-25T20:51:00.145 INFO:teuthology.orchestra.run.smithi174.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-12-25T20:51:01.163 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 8 - Extras 48 kB/s | 18 kB 00:00 2023-12-25T20:51:01.805 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 8 - Extras common packages 16 kB/s | 6.9 kB 00:00 2023-12-25T20:51:02.131 INFO:teuthology.orchestra.run.smithi061.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-12-25T20:51:02.311 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 16 MB 00:02 2023-12-25T20:51:03.665 INFO:teuthology.orchestra.run.smithi061.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2023-12-25T20:51:06.144 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 395 kB/s | 24 kB 00:00 2023-12-25T20:51:07.807 INFO:teuthology.orchestra.run.smithi061.stdout:lab-extras 296 kB/s | 24 kB 00:00 2023-12-25T20:51:09.852 INFO:teuthology.orchestra.run.smithi174.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-25T20:51:09.852 INFO:teuthology.orchestra.run.smithi174.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-25T20:51:10.150 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T20:51:10.150 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T20:51:10.150 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2023-12-25T20:51:10.150 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T20:51:10.151 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2023-12-25T20:51:10.151 INFO:teuthology.orchestra.run.smithi174.stdout: kernel x86_64 4.18.0-529.el8 baseos 10 M 2023-12-25T20:51:10.151 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2023-12-25T20:51:10.151 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-core x86_64 4.18.0-529.el8 baseos 43 M 2023-12-25T20:51:10.151 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules x86_64 4.18.0-529.el8 baseos 36 M 2023-12-25T20:51:10.151 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T20:51:10.151 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2023-12-25T20:51:10.151 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T20:51:10.151 INFO:teuthology.orchestra.run.smithi174.stdout:Install 3 Packages 2023-12-25T20:51:10.151 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T20:51:10.152 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 89 M 2023-12-25T20:51:10.152 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 96 M 2023-12-25T20:51:10.152 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2023-12-25T20:51:11.774 INFO:teuthology.orchestra.run.smithi061.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-25T20:51:11.774 INFO:teuthology.orchestra.run.smithi061.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-25T20:51:12.096 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T20:51:12.096 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T20:51:12.096 INFO:teuthology.orchestra.run.smithi061.stdout: Package Architecture Version Repository Size 2023-12-25T20:51:12.097 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T20:51:12.097 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-12-25T20:51:12.097 INFO:teuthology.orchestra.run.smithi061.stdout: kernel x86_64 4.18.0-529.el8 baseos 10 M 2023-12-25T20:51:12.097 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-12-25T20:51:12.097 INFO:teuthology.orchestra.run.smithi061.stdout: kernel-core x86_64 4.18.0-529.el8 baseos 43 M 2023-12-25T20:51:12.097 INFO:teuthology.orchestra.run.smithi061.stdout: kernel-modules x86_64 4.18.0-529.el8 baseos 36 M 2023-12-25T20:51:12.097 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T20:51:12.097 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-12-25T20:51:12.097 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T20:51:12.097 INFO:teuthology.orchestra.run.smithi061.stdout:Install 3 Packages 2023-12-25T20:51:12.097 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T20:51:12.098 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 89 M 2023-12-25T20:51:12.098 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 96 M 2023-12-25T20:51:12.098 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-12-25T20:51:13.300 INFO:teuthology.orchestra.run.smithi061.stdout:(1/3): kernel-4.18.0-529.el8.x86_64.rpm 9.2 MB/s | 10 MB 00:01 2023-12-25T20:51:13.587 INFO:teuthology.orchestra.run.smithi174.stdout:(1/3): kernel-4.18.0-529.el8.x86_64.rpm 3.1 MB/s | 10 MB 00:03 2023-12-25T20:51:16.634 INFO:teuthology.orchestra.run.smithi061.stdout:(2/3): kernel-modules-4.18.0-529.el8.x86_64.rpm 8.1 MB/s | 36 MB 00:04 2023-12-25T20:51:16.870 INFO:teuthology.orchestra.run.smithi174.stdout:(2/3): kernel-core-4.18.0-529.el8.x86_64.rpm 6.6 MB/s | 43 MB 00:06 2023-12-25T20:51:20.826 INFO:teuthology.orchestra.run.smithi061.stdout:(3/3): kernel-core-4.18.0-529.el8.x86_64.rpm 5.0 MB/s | 43 MB 00:08 2023-12-25T20:51:20.826 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:51:20.826 INFO:teuthology.orchestra.run.smithi061.stdout:Total 10 MB/s | 89 MB 00:08 2023-12-25T20:51:21.533 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-12-25T20:51:21.997 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-12-25T20:51:21.997 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-12-25T20:51:22.311 INFO:teuthology.orchestra.run.smithi174.stdout:(3/3): kernel-modules-4.18.0-529.el8.x86_64.rpm 3.0 MB/s | 36 MB 00:11 2023-12-25T20:51:22.311 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2023-12-25T20:51:22.312 INFO:teuthology.orchestra.run.smithi174.stdout:Total 7.3 MB/s | 89 MB 00:12 2023-12-25T20:51:22.989 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2023-12-25T20:51:23.290 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-12-25T20:51:23.320 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-12-25T20:51:23.402 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2023-12-25T20:51:23.402 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2023-12-25T20:51:24.656 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2023-12-25T20:51:24.686 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2023-12-25T20:51:26.263 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-12-25T20:51:26.841 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-25T20:51:27.469 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2023-12-25T20:51:28.016 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-25T20:51:30.543 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-25T20:51:30.924 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-25T20:51:31.627 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-25T20:51:32.007 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-25T20:51:34.597 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-25T20:51:34.909 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : kernel-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:51:35.431 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-25T20:51:35.729 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:51:57.600 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:51:57.624 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:51:57.966 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:51:57.966 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-4.18.0-529.el8.x86_64 1/3 2023-12-25T20:51:57.966 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-core-4.18.0-529.el8.x86_64 2/3 2023-12-25T20:51:58.250 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:51:58.275 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:51:58.765 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: kernel-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:51:58.765 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : kernel-4.18.0-529.el8.x86_64 1/3 2023-12-25T20:51:58.765 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : kernel-core-4.18.0-529.el8.x86_64 2/3 2023-12-25T20:51:58.768 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:51:58.768 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T20:51:58.768 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2023-12-25T20:51:58.768 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-4.18.0-529.el8.x86_64 kernel-core-4.18.0-529.el8.x86_64 2023-12-25T20:51:58.768 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules-4.18.0-529.el8.x86_64 2023-12-25T20:51:58.768 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T20:51:58.768 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T20:51:58.919 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q kernel | sort -rV | head -n 1 2023-12-25T20:51:59.137 INFO:teuthology.orchestra.run.smithi174.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-25T20:51:59.137 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-25T20:51:59.137 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-529.el8.x86_64 2023-12-25T20:51:59.137 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-25T20:51:59.138 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-25T20:51:59.138 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-12-25T20:51:59.138 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi174.front.sepia.ceph.com, path=None, version=distro) 2023-12-25T20:51:59.138 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q kernel | sort -rV | head -n 1 2023-12-25T20:51:59.347 INFO:teuthology.orchestra.run.smithi174.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-25T20:51:59.347 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-25T20:51:59.347 DEBUG:teuthology.orchestra.run.smithi174:> sudo rpm -qi grub2-tools 2023-12-25T20:51:59.380 INFO:teuthology.orchestra.run.smithi174.stdout:Name : grub2-tools 2023-12-25T20:51:59.380 INFO:teuthology.orchestra.run.smithi174.stdout:Epoch : 1 2023-12-25T20:51:59.381 INFO:teuthology.orchestra.run.smithi174.stdout:Version : 2.02 2023-12-25T20:51:59.381 INFO:teuthology.orchestra.run.smithi174.stdout:Release : 129.el8 2023-12-25T20:51:59.381 INFO:teuthology.orchestra.run.smithi174.stdout:Architecture: x86_64 2023-12-25T20:51:59.381 INFO:teuthology.orchestra.run.smithi174.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-25T20:51:59.381 INFO:teuthology.orchestra.run.smithi174.stdout:Group : System Environment/Base 2023-12-25T20:51:59.381 INFO:teuthology.orchestra.run.smithi174.stdout:Size : 9265586 2023-12-25T20:51:59.381 INFO:teuthology.orchestra.run.smithi174.stdout:License : GPLv3+ 2023-12-25T20:51:59.381 INFO:teuthology.orchestra.run.smithi174.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-25T20:51:59.381 INFO:teuthology.orchestra.run.smithi174.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-25T20:51:59.381 INFO:teuthology.orchestra.run.smithi174.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-25T20:51:59.381 INFO:teuthology.orchestra.run.smithi174.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-25T20:51:59.381 INFO:teuthology.orchestra.run.smithi174.stdout:Relocations : (not relocatable) 2023-12-25T20:51:59.381 INFO:teuthology.orchestra.run.smithi174.stdout:Packager : CentOS BuildSystem 2023-12-25T20:51:59.382 INFO:teuthology.orchestra.run.smithi174.stdout:Vendor : CentOS 2023-12-25T20:51:59.382 INFO:teuthology.orchestra.run.smithi174.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-25T20:51:59.382 INFO:teuthology.orchestra.run.smithi174.stdout:Summary : Support tools for GRUB. 2023-12-25T20:51:59.382 INFO:teuthology.orchestra.run.smithi174.stdout:Description : 2023-12-25T20:51:59.382 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T20:51:59.382 INFO:teuthology.orchestra.run.smithi174.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-25T20:51:59.382 INFO:teuthology.orchestra.run.smithi174.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-25T20:51:59.382 INFO:teuthology.orchestra.run.smithi174.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-25T20:51:59.382 INFO:teuthology.orchestra.run.smithi174.stdout:hardware devices. 2023-12-25T20:51:59.382 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T20:51:59.382 INFO:teuthology.orchestra.run.smithi174.stdout:This subpackage provides tools for support of all platforms. 2023-12-25T20:51:59.383 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-25T20:51:59.383 INFO:teuthology.task.kernel:Updating grub on smithi174 to boot 4.18.0-529.el8.x86_64 2023-12-25T20:51:59.383 DEBUG:teuthology.orchestra.run.smithi174:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-25T20:51:59.596 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-25T20:51:59.596 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T20:51:59.596 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-12-25T20:51:59.596 INFO:teuthology.orchestra.run.smithi061.stdout: kernel-4.18.0-529.el8.x86_64 kernel-core-4.18.0-529.el8.x86_64 2023-12-25T20:51:59.596 INFO:teuthology.orchestra.run.smithi061.stdout: kernel-modules-4.18.0-529.el8.x86_64 2023-12-25T20:51:59.596 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T20:51:59.596 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T20:51:59.827 DEBUG:teuthology.orchestra.run.smithi061:> rpm -q kernel | sort -rV | head -n 1 2023-12-25T20:52:00.100 INFO:teuthology.orchestra.run.smithi061.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-25T20:52:00.100 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-25T20:52:00.100 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-529.el8.x86_64 2023-12-25T20:52:00.100 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-25T20:52:00.100 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-25T20:52:00.100 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-12-25T20:52:00.101 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi061.front.sepia.ceph.com, path=None, version=distro) 2023-12-25T20:52:00.101 DEBUG:teuthology.orchestra.run.smithi061:> rpm -q kernel | sort -rV | head -n 1 2023-12-25T20:52:00.319 INFO:teuthology.orchestra.run.smithi061.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-25T20:52:00.320 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-25T20:52:00.320 DEBUG:teuthology.orchestra.run.smithi061:> sudo rpm -qi grub2-tools 2023-12-25T20:52:00.354 INFO:teuthology.orchestra.run.smithi061.stdout:Name : grub2-tools 2023-12-25T20:52:00.354 INFO:teuthology.orchestra.run.smithi061.stdout:Epoch : 1 2023-12-25T20:52:00.354 INFO:teuthology.orchestra.run.smithi061.stdout:Version : 2.02 2023-12-25T20:52:00.354 INFO:teuthology.orchestra.run.smithi061.stdout:Release : 129.el8 2023-12-25T20:52:00.354 INFO:teuthology.orchestra.run.smithi061.stdout:Architecture: x86_64 2023-12-25T20:52:00.355 INFO:teuthology.orchestra.run.smithi061.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-25T20:52:00.355 INFO:teuthology.orchestra.run.smithi061.stdout:Group : System Environment/Base 2023-12-25T20:52:00.355 INFO:teuthology.orchestra.run.smithi061.stdout:Size : 9265586 2023-12-25T20:52:00.355 INFO:teuthology.orchestra.run.smithi061.stdout:License : GPLv3+ 2023-12-25T20:52:00.355 INFO:teuthology.orchestra.run.smithi061.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-25T20:52:00.355 INFO:teuthology.orchestra.run.smithi061.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-25T20:52:00.355 INFO:teuthology.orchestra.run.smithi061.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-25T20:52:00.355 INFO:teuthology.orchestra.run.smithi061.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-25T20:52:00.355 INFO:teuthology.orchestra.run.smithi061.stdout:Relocations : (not relocatable) 2023-12-25T20:52:00.355 INFO:teuthology.orchestra.run.smithi061.stdout:Packager : CentOS BuildSystem 2023-12-25T20:52:00.355 INFO:teuthology.orchestra.run.smithi061.stdout:Vendor : CentOS 2023-12-25T20:52:00.355 INFO:teuthology.orchestra.run.smithi061.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-25T20:52:00.355 INFO:teuthology.orchestra.run.smithi061.stdout:Summary : Support tools for GRUB. 2023-12-25T20:52:00.355 INFO:teuthology.orchestra.run.smithi061.stdout:Description : 2023-12-25T20:52:00.356 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T20:52:00.356 INFO:teuthology.orchestra.run.smithi061.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-25T20:52:00.356 INFO:teuthology.orchestra.run.smithi061.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-25T20:52:00.356 INFO:teuthology.orchestra.run.smithi061.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-25T20:52:00.356 INFO:teuthology.orchestra.run.smithi061.stdout:hardware devices. 2023-12-25T20:52:00.356 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T20:52:00.356 INFO:teuthology.orchestra.run.smithi061.stdout:This subpackage provides tools for support of all platforms. 2023-12-25T20:52:00.357 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-25T20:52:00.357 INFO:teuthology.task.kernel:Updating grub on smithi061 to boot 4.18.0-529.el8.x86_64 2023-12-25T20:52:00.357 DEBUG:teuthology.orchestra.run.smithi061:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-25T20:52:01.011 INFO:teuthology.orchestra.run.smithi174.stderr:Generating grub configuration file ... 2023-12-25T20:52:01.734 INFO:teuthology.orchestra.run.smithi061.stderr:Generating grub configuration file ... 2023-12-25T20:52:03.662 INFO:teuthology.orchestra.run.smithi174.stderr:done 2023-12-25T20:52:03.663 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2023-12-25T20:52:03.665 INFO:teuthology.orchestra.run.smithi061.stderr:done 2023-12-25T20:52:03.667 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2023-12-25T20:52:03.688 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.7YVqOvCuHW 2023-12-25T20:52:03.688 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.7YVqOvCuHW 2023-12-25T20:52:03.697 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.BV2lIaktOZ 2023-12-25T20:52:03.697 DEBUG:teuthology.orchestra.run.smithi174:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.BV2lIaktOZ 2023-12-25T20:52:03.771 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.BV2lIaktOZ 2023-12-25T20:52:03.773 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.7YVqOvCuHW 2023-12-25T20:52:03.919 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.BV2lIaktOZ is 6KB 2023-12-25T20:52:03.919 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.7YVqOvCuHW is 6KB 2023-12-25T20:52:03.968 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.BV2lIaktOZ 2023-12-25T20:52:03.972 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.7YVqOvCuHW 2023-12-25T20:52:03.986 DEBUG:teuthology.orchestra.run.smithi174:> sudo /bin/ls /boot/loader/entries || true 2023-12-25T20:52:03.990 DEBUG:teuthology.orchestra.run.smithi061:> sudo /bin/ls /boot/loader/entries || true 2023-12-25T20:52:04.052 INFO:teuthology.orchestra.run.smithi174.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-25T20:52:04.052 INFO:teuthology.orchestra.run.smithi174.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-25T20:52:04.052 INFO:teuthology.orchestra.run.smithi174.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-25T20:52:04.052 INFO:teuthology.orchestra.run.smithi174.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64.conf 2023-12-25T20:52:04.053 DEBUG:teuthology.orchestra.run.smithi174:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64 2023-12-25T20:52:04.056 INFO:teuthology.orchestra.run.smithi061.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-25T20:52:04.056 INFO:teuthology.orchestra.run.smithi061.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-25T20:52:04.056 INFO:teuthology.orchestra.run.smithi061.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-25T20:52:04.056 INFO:teuthology.orchestra.run.smithi061.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64.conf 2023-12-25T20:52:04.057 DEBUG:teuthology.orchestra.run.smithi061:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64 2023-12-25T20:52:04.180 DEBUG:teuthology.orchestra.run.smithi061:> sudo shutdown -r now 2023-12-25T20:52:04.218 DEBUG:teuthology.orchestra.run.smithi174:> sudo shutdown -r now 2023-12-25T20:52:34.209 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-12-25T20:52:34.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-12-25T20:52:34.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T20:52:34.221 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-12-25T20:52:34.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2023-12-25T20:52:34.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T20:53:08.754 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-12-25T20:53:17.761 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-12-25T20:53:17.762 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T20:53:34.240 DEBUG:teuthology.orchestra.remote:timed out 2023-12-25T20:53:36.149 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-12-25T20:53:43.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2023-12-25T20:53:43.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T20:53:43.623 DEBUG:teuthology.orchestra.run.smithi174:> true 2023-12-25T20:53:43.926 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2023-12-25T20:53:43.926 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-529.el8.x86_64"... 2023-12-25T20:53:43.927 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2023-12-25T20:53:43.943 INFO:teuthology.orchestra.run.smithi174.stdout:4.18.0-529.el8.x86_64 2023-12-25T20:53:43.944 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-529.el8.x86_64 vs 4.18.0-529.el8.x86_64 2023-12-25T20:53:43.944 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-25T20:53:43.944 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-25T20:53:44.945 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-12-25T20:53:44.946 DEBUG:teuthology.orchestra.run.smithi174:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-25T20:53:45.031 INFO:teuthology.orchestra.run.smithi174.stdout:ttyS1 2023-12-25T20:53:45.047 DEBUG:teuthology.parallel:result is None 2023-12-25T20:53:48.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-12-25T20:53:48.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T20:53:48.521 DEBUG:teuthology.orchestra.run.smithi061:> true 2023-12-25T20:53:48.780 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-12-25T20:53:48.780 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-529.el8.x86_64"... 2023-12-25T20:53:48.780 DEBUG:teuthology.orchestra.run.smithi061:> uname -r 2023-12-25T20:53:48.843 INFO:teuthology.orchestra.run.smithi061.stdout:4.18.0-529.el8.x86_64 2023-12-25T20:53:48.843 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-529.el8.x86_64 vs 4.18.0-529.el8.x86_64 2023-12-25T20:53:48.843 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-25T20:53:48.843 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-25T20:53:49.844 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-12-25T20:53:49.844 DEBUG:teuthology.orchestra.run.smithi061:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-25T20:53:49.924 INFO:teuthology.orchestra.run.smithi061.stdout:ttyS1 2023-12-25T20:53:49.937 DEBUG:teuthology.parallel:result is None 2023-12-25T20:53:49.937 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-25T20:53:49.947 INFO:teuthology.task.internal:Creating test directory... 2023-12-25T20:53:49.947 DEBUG:teuthology.orchestra.run.smithi061:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-25T20:53:49.949 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-25T20:53:49.967 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-25T20:53:49.973 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-25T20:53:49.981 INFO:teuthology.task.internal:Creating archive directory... 2023-12-25T20:53:49.981 DEBUG:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-25T20:53:50.012 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-25T20:53:50.046 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-25T20:53:50.054 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-25T20:53:50.054 DEBUG:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-25T20:53:50.074 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-25T20:53:50.102 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-25T20:53:50.111 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-25T20:53:50.114 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-25T20:53:50.124 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-25T20:53:50.126 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-25T20:53:50.133 INFO:teuthology.task.internal:Configuring sudo... 2023-12-25T20:53:50.133 DEBUG:teuthology.orchestra.run.smithi061:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-25T20:53:50.158 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-25T20:53:50.196 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-25T20:53:50.205 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-25T20:53:50.205 DEBUG:teuthology.orchestra.run.smithi061:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-25T20:53:50.229 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-25T20:53:50.258 DEBUG:teuthology.orchestra.run.smithi061:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-25T20:53:50.334 DEBUG:teuthology.orchestra.run.smithi061:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-25T20:53:50.401 DEBUG:teuthology.orchestra.run.smithi061:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-25T20:53:50.447 DEBUG:teuthology.orchestra.run.smithi061:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-25T20:53:50.491 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-12-25T20:53:50.492 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-25T20:53:50.560 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-25T20:53:50.606 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-25T20:53:50.666 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-25T20:53:50.756 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-25T20:53:50.800 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-12-25T20:53:50.801 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-25T20:53:50.871 DEBUG:teuthology.orchestra.run.smithi061:> sudo service rsyslog restart 2023-12-25T20:53:50.873 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2023-12-25T20:53:50.941 INFO:teuthology.orchestra.run.smithi061.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-25T20:53:50.993 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-25T20:53:51.495 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-25T20:53:51.502 INFO:teuthology.task.internal:Starting timer... 2023-12-25T20:53:51.503 INFO:teuthology.run_tasks:Running task pcp... 2023-12-25T20:53:51.515 INFO:teuthology.run_tasks:Running task selinux... 2023-12-25T20:53:51.524 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-25T20:53:51.525 DEBUG:teuthology.orchestra.run.smithi061:> sudo service auditd rotate 2023-12-25T20:53:52.254 INFO:teuthology.orchestra.run.smithi061.stdout:Rotating logs: 2023-12-25T20:53:52.280 DEBUG:teuthology.orchestra.run.smithi174:> sudo service auditd rotate 2023-12-25T20:53:52.352 INFO:teuthology.orchestra.run.smithi174.stdout:Rotating logs: 2023-12-25T20:53:52.354 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-25T20:53:52.354 DEBUG:teuthology.orchestra.run.smithi061:> /usr/sbin/getenforce 2023-12-25T20:53:52.785 INFO:teuthology.orchestra.run.smithi061.stdout:Permissive 2023-12-25T20:53:52.785 DEBUG:teuthology.orchestra.run.smithi174:> /usr/sbin/getenforce 2023-12-25T20:53:52.843 INFO:teuthology.orchestra.run.smithi174.stdout:Permissive 2023-12-25T20:53:52.843 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi061.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi174.front.sepia.ceph.com': 'permissive'} 2023-12-25T20:53:52.843 DEBUG:teuthology.orchestra.run.smithi061:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-25T20:53:52.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T20:53:52.874 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-25T20:53:52.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T20:53:52.901 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-25T20:53:52.901 DEBUG:teuthology.orchestra.run.smithi061:> sudo /usr/sbin/setenforce permissive 2023-12-25T20:53:52.949 DEBUG:teuthology.orchestra.run.smithi174:> sudo /usr/sbin/setenforce permissive 2023-12-25T20:53:52.977 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-25T20:53:52.992 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-25T20:53:52.998 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2023-12-25T20:53:53.153 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-25T20:53:53.176 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-25T20:53:53.177 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi061.front.sepia.ceph.com,smithi174.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-25T21:01:35.935 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi061.front.sepia.ceph.com'), Remote(name='ubuntu@smithi174.front.sepia.ceph.com')] 2023-12-25T21:01:35.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-12-25T21:01:35.937 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T21:01:36.005 DEBUG:teuthology.orchestra.run.smithi061:> true 2023-12-25T21:01:36.072 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-12-25T21:01:36.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2023-12-25T21:01:36.073 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-25T21:01:36.140 DEBUG:teuthology.orchestra.run.smithi174:> true 2023-12-25T21:01:36.207 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2023-12-25T21:01:36.207 INFO:teuthology.run_tasks:Running task clock... 2023-12-25T21:01:36.219 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-25T21:01:36.219 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-25T21:01:36.220 DEBUG:teuthology.orchestra.run.smithi061:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-25T21:01:36.222 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-25T21:01:36.222 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-25T21:01:36.254 INFO:teuthology.orchestra.run.smithi061.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-25T21:01:36.269 INFO:teuthology.orchestra.run.smithi061.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-25T21:01:36.277 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-25T21:01:36.290 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-25T21:01:36.353 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: ntpd: command not found 2023-12-25T21:01:36.354 INFO:teuthology.orchestra.run.smithi061.stderr:sudo: ntpd: command not found 2023-12-25T21:01:36.364 INFO:teuthology.orchestra.run.smithi174.stdout:506 Cannot talk to daemon 2023-12-25T21:01:36.366 INFO:teuthology.orchestra.run.smithi061.stdout:506 Cannot talk to daemon 2023-12-25T21:01:36.378 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-25T21:01:36.380 INFO:teuthology.orchestra.run.smithi061.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-25T21:01:36.391 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-25T21:01:36.394 INFO:teuthology.orchestra.run.smithi061.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-25T21:01:36.454 INFO:teuthology.orchestra.run.smithi174.stderr:bash: ntpq: command not found 2023-12-25T21:01:36.457 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-25T21:01:36.458 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2023-12-25T21:01:36.458 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T21:01:36.458 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T21:01:36.458 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T21:01:36.458 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T21:01:36.479 INFO:teuthology.orchestra.run.smithi061.stderr:bash: ntpq: command not found 2023-12-25T21:01:36.482 INFO:teuthology.orchestra.run.smithi061.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-25T21:01:36.482 INFO:teuthology.orchestra.run.smithi061.stdout:=============================================================================== 2023-12-25T21:01:36.482 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T21:01:36.483 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T21:01:36.483 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T21:01:36.483 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T21:01:36.484 INFO:teuthology.run_tasks:Running task pexec... 2023-12-25T21:01:36.493 INFO:teuthology.task.pexec:Executing custom commands... 2023-12-25T21:01:36.494 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi061.front.sepia.ceph.com 2023-12-25T21:01:36.494 DEBUG:teuthology.orchestra.run.smithi061:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-25T21:01:36.494 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi174.front.sepia.ceph.com 2023-12-25T21:01:36.494 DEBUG:teuthology.orchestra.run.smithi174:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-25T21:01:37.379 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:52 ago on Mon 25 Dec 2023 08:59:45 PM UTC. 2023-12-25T21:01:37.435 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:42 ago on Mon 25 Dec 2023 08:59:55 PM UTC. 2023-12-25T21:01:39.494 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:01:39.494 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:01:39.495 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2023-12-25T21:01:39.495 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:01:39.495 INFO:teuthology.orchestra.run.smithi174.stdout:Resetting modules: 2023-12-25T21:01:39.495 INFO:teuthology.orchestra.run.smithi174.stdout: container-tools 2023-12-25T21:01:39.495 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:01:39.495 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2023-12-25T21:01:39.495 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:01:39.495 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:01:39.748 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:01:39.769 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:01:39.770 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:01:39.770 INFO:teuthology.orchestra.run.smithi061.stdout: Package Architecture Version Repository Size 2023-12-25T21:01:39.770 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:01:39.770 INFO:teuthology.orchestra.run.smithi061.stdout:Resetting modules: 2023-12-25T21:01:39.770 INFO:teuthology.orchestra.run.smithi061.stdout: container-tools 2023-12-25T21:01:39.770 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:01:39.770 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-12-25T21:01:39.771 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:01:39.771 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:01:40.035 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:01:40.679 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:55 ago on Mon 25 Dec 2023 08:59:45 PM UTC. 2023-12-25T21:01:41.009 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:45 ago on Mon 25 Dec 2023 08:59:55 PM UTC. 2023-12-25T21:01:42.818 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:01:43.008 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:01:43.008 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2023-12-25T21:01:43.008 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:01:43.009 INFO:teuthology.orchestra.run.smithi174.stdout:Installing group/module packages: 2023-12-25T21:01:43.009 INFO:teuthology.orchestra.run.smithi174.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-12-25T21:01:43.009 INFO:teuthology.orchestra.run.smithi174.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-12-25T21:01:43.009 INFO:teuthology.orchestra.run.smithi174.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-12-25T21:01:43.009 INFO:teuthology.orchestra.run.smithi174.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-12-25T21:01:43.009 INFO:teuthology.orchestra.run.smithi174.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-12-25T21:01:43.009 INFO:teuthology.orchestra.run.smithi174.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-12-25T21:01:43.010 INFO:teuthology.orchestra.run.smithi174.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-12-25T21:01:43.010 INFO:teuthology.orchestra.run.smithi174.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-12-25T21:01:43.010 INFO:teuthology.orchestra.run.smithi174.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-12-25T21:01:43.010 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2023-12-25T21:01:43.010 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-12-25T21:01:43.010 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-12-25T21:01:43.010 INFO:teuthology.orchestra.run.smithi174.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-12-25T21:01:43.010 INFO:teuthology.orchestra.run.smithi174.stdout:Installing weak dependencies: 2023-12-25T21:01:43.010 INFO:teuthology.orchestra.run.smithi174.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-12-25T21:01:43.010 INFO:teuthology.orchestra.run.smithi174.stdout:Installing module profiles: 2023-12-25T21:01:43.010 INFO:teuthology.orchestra.run.smithi174.stdout: container-tools/common 2023-12-25T21:01:43.011 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:01:43.011 INFO:teuthology.orchestra.run.smithi174.stdout:Enabling module streams: 2023-12-25T21:01:43.011 INFO:teuthology.orchestra.run.smithi174.stdout: container-tools rhel8 2023-12-25T21:01:43.011 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:01:43.011 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2023-12-25T21:01:43.011 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:01:43.011 INFO:teuthology.orchestra.run.smithi174.stdout:Install 13 Packages 2023-12-25T21:01:43.011 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:01:43.012 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 27 M 2023-12-25T21:01:43.012 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 89 M 2023-12-25T21:01:43.012 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2023-12-25T21:01:43.274 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:01:43.478 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:01:43.478 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-12-25T21:01:43.479 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:01:43.479 INFO:teuthology.orchestra.run.smithi061.stdout:Installing group/module packages: 2023-12-25T21:01:43.479 INFO:teuthology.orchestra.run.smithi061.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-12-25T21:01:43.479 INFO:teuthology.orchestra.run.smithi061.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-12-25T21:01:43.479 INFO:teuthology.orchestra.run.smithi061.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-12-25T21:01:43.479 INFO:teuthology.orchestra.run.smithi061.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-12-25T21:01:43.479 INFO:teuthology.orchestra.run.smithi061.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-12-25T21:01:43.479 INFO:teuthology.orchestra.run.smithi061.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-12-25T21:01:43.479 INFO:teuthology.orchestra.run.smithi061.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-12-25T21:01:43.479 INFO:teuthology.orchestra.run.smithi061.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-12-25T21:01:43.479 INFO:teuthology.orchestra.run.smithi061.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-12-25T21:01:43.479 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-12-25T21:01:43.479 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-12-25T21:01:43.479 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-12-25T21:01:43.480 INFO:teuthology.orchestra.run.smithi061.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-12-25T21:01:43.480 INFO:teuthology.orchestra.run.smithi061.stdout:Installing weak dependencies: 2023-12-25T21:01:43.480 INFO:teuthology.orchestra.run.smithi061.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-12-25T21:01:43.480 INFO:teuthology.orchestra.run.smithi061.stdout:Installing module profiles: 2023-12-25T21:01:43.480 INFO:teuthology.orchestra.run.smithi061.stdout: container-tools/common 2023-12-25T21:01:43.480 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:01:43.480 INFO:teuthology.orchestra.run.smithi061.stdout:Enabling module streams: 2023-12-25T21:01:43.480 INFO:teuthology.orchestra.run.smithi061.stdout: container-tools rhel8 2023-12-25T21:01:43.481 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:01:43.481 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-12-25T21:01:43.481 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:01:43.481 INFO:teuthology.orchestra.run.smithi061.stdout:Install 13 Packages 2023-12-25T21:01:43.481 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:01:43.481 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 27 M 2023-12-25T21:01:43.481 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 89 M 2023-12-25T21:01:43.481 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-12-25T21:01:43.755 INFO:teuthology.orchestra.run.smithi174.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 1.3 MB/s | 750 kB 00:00 2023-12-25T21:01:43.788 INFO:teuthology.orchestra.run.smithi174.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 1.7 MB/s | 1.0 MB 00:00 2023-12-25T21:01:43.863 INFO:teuthology.orchestra.run.smithi174.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 349 kB/s | 38 kB 00:00 2023-12-25T21:01:43.906 INFO:teuthology.orchestra.run.smithi174.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.0 MB/s | 241 kB 00:00 2023-12-25T21:01:43.998 INFO:teuthology.orchestra.run.smithi174.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.9 MB/s | 178 kB 00:00 2023-12-25T21:01:44.115 INFO:teuthology.orchestra.run.smithi174.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 212 kB/s | 25 kB 00:00 2023-12-25T21:01:44.182 INFO:teuthology.orchestra.run.smithi174.stdout:(7/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 12 MB/s | 3.9 MB 00:00 2023-12-25T21:01:44.232 INFO:teuthology.orchestra.run.smithi174.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 803 kB/s | 94 kB 00:00 2023-12-25T21:01:44.413 INFO:teuthology.orchestra.run.smithi061.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 1.0 MB/s | 750 kB 00:00 2023-12-25T21:01:44.457 INFO:teuthology.orchestra.run.smithi174.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 7.6 MB/s | 9.6 MB 00:01 2023-12-25T21:01:44.508 INFO:teuthology.orchestra.run.smithi174.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 9.2 MB/s | 2.5 MB 00:00 2023-12-25T21:01:44.530 INFO:teuthology.orchestra.run.smithi061.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 324 kB/s | 38 kB 00:00 2023-12-25T21:01:44.549 INFO:teuthology.orchestra.run.smithi174.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 573 kB/s | 53 kB 00:00 2023-12-25T21:01:44.600 INFO:teuthology.orchestra.run.smithi174.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 447 kB/s | 41 kB 00:00 2023-12-25T21:01:44.700 INFO:teuthology.orchestra.run.smithi174.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 16 MB/s | 8.1 MB 00:00 2023-12-25T21:01:44.701 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:01:44.701 INFO:teuthology.orchestra.run.smithi174.stdout:Total 16 MB/s | 27 MB 00:01 2023-12-25T21:01:44.704 INFO:teuthology.orchestra.run.smithi061.stdout:(3/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 1.3 MB/s | 241 kB 00:00 2023-12-25T21:01:44.738 INFO:teuthology.orchestra.run.smithi061.stdout:(4/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 955 kB/s | 1.0 MB 00:01 2023-12-25T21:01:44.905 INFO:teuthology.orchestra.run.smithi061.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.0 MB/s | 178 kB 00:00 2023-12-25T21:01:44.968 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2023-12-25T21:01:44.989 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2023-12-25T21:01:44.989 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2023-12-25T21:01:45.127 INFO:teuthology.orchestra.run.smithi061.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 112 kB/s | 25 kB 00:00 2023-12-25T21:01:45.223 INFO:teuthology.orchestra.run.smithi061.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 991 kB/s | 94 kB 00:00 2023-12-25T21:01:45.252 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2023-12-25T21:01:45.255 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2023-12-25T21:01:45.950 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2023-12-25T21:01:46.081 INFO:teuthology.orchestra.run.smithi061.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 2.8 MB/s | 3.9 MB 00:01 2023-12-25T21:01:46.224 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-12-25T21:01:46.394 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-12-25T21:01:46.623 INFO:teuthology.orchestra.run.smithi061.stdout:(9/13): toolbox-0.0.99.4-6.module_el8+712+4cd1b 4.7 MB/s | 2.5 MB 00:00 2023-12-25T21:01:46.630 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-12-25T21:01:46.723 INFO:teuthology.orchestra.run.smithi061.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 527 kB/s | 53 kB 00:00 2023-12-25T21:01:46.815 INFO:teuthology.orchestra.run.smithi061.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 448 kB/s | 41 kB 00:00 2023-12-25T21:01:46.847 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-25T21:01:47.015 INFO:teuthology.orchestra.run.smithi061.stdout:(12/13): buildah-1.32.0-1.module_el8+712+4cd1bd 2.9 MB/s | 9.6 MB 00:03 2023-12-25T21:01:47.405 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-12-25T21:01:47.599 INFO:teuthology.orchestra.run.smithi061.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 3.4 MB/s | 8.1 MB 00:02 2023-12-25T21:01:47.600 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:01:47.600 INFO:teuthology.orchestra.run.smithi061.stdout:Total 6.4 MB/s | 27 MB 00:04 2023-12-25T21:01:47.879 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-12-25T21:01:47.902 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-12-25T21:01:47.902 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-12-25T21:01:48.161 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-12-25T21:01:48.164 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-12-25T21:01:48.386 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-25T21:01:48.611 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-12-25T21:01:48.841 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-12-25T21:01:48.871 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-12-25T21:01:49.065 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-12-25T21:01:49.089 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-12-25T21:01:49.275 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-12-25T21:01:49.490 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-12-25T21:01:49.500 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-12-25T21:01:49.757 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-25T21:01:50.333 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-12-25T21:01:50.412 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-25T21:01:50.664 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-12-25T21:01:50.851 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-25T21:01:51.277 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-25T21:01:51.277 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-12-25T21:01:51.277 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-12-25T21:01:51.277 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-12-25T21:01:51.278 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-25T21:01:51.278 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-12-25T21:01:51.278 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-25T21:01:51.278 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-12-25T21:01:51.278 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-12-25T21:01:51.278 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-12-25T21:01:51.278 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-12-25T21:01:51.278 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-25T21:01:51.278 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-12-25T21:01:51.378 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-25T21:01:51.652 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-12-25T21:01:51.877 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-12-25T21:01:52.113 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-12-25T21:01:52.135 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-12-25T21:01:52.135 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:01:52.135 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2023-12-25T21:01:52.135 INFO:teuthology.orchestra.run.smithi174.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-25T21:01:52.135 INFO:teuthology.orchestra.run.smithi174.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-25T21:01:52.135 INFO:teuthology.orchestra.run.smithi174.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-12-25T21:01:52.135 INFO:teuthology.orchestra.run.smithi174.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-12-25T21:01:52.136 INFO:teuthology.orchestra.run.smithi174.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-12-25T21:01:52.136 INFO:teuthology.orchestra.run.smithi174.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-12-25T21:01:52.136 INFO:teuthology.orchestra.run.smithi174.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-12-25T21:01:52.136 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-12-25T21:01:52.136 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-12-25T21:01:52.136 INFO:teuthology.orchestra.run.smithi174.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-12-25T21:01:52.136 INFO:teuthology.orchestra.run.smithi174.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-12-25T21:01:52.136 INFO:teuthology.orchestra.run.smithi174.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-12-25T21:01:52.136 INFO:teuthology.orchestra.run.smithi174.stdout: yajl-2.1.0-12.el8.x86_64 2023-12-25T21:01:52.136 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:01:52.136 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:01:52.440 DEBUG:teuthology.parallel:result is None 2023-12-25T21:01:52.549 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-12-25T21:01:54.101 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-25T21:01:54.330 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-12-25T21:01:54.515 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-25T21:01:54.967 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-25T21:01:54.967 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-12-25T21:01:54.968 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-12-25T21:01:54.968 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-12-25T21:01:54.968 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-25T21:01:54.968 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-12-25T21:01:54.968 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-25T21:01:54.968 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-12-25T21:01:54.968 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-12-25T21:01:54.968 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-12-25T21:01:54.968 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-12-25T21:01:54.968 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-25T21:01:54.968 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-12-25T21:01:55.820 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-12-25T21:01:55.820 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:01:55.820 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-12-25T21:01:55.820 INFO:teuthology.orchestra.run.smithi061.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-25T21:01:55.820 INFO:teuthology.orchestra.run.smithi061.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-25T21:01:55.820 INFO:teuthology.orchestra.run.smithi061.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-12-25T21:01:55.820 INFO:teuthology.orchestra.run.smithi061.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-12-25T21:01:55.820 INFO:teuthology.orchestra.run.smithi061.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-12-25T21:01:55.820 INFO:teuthology.orchestra.run.smithi061.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-12-25T21:01:55.820 INFO:teuthology.orchestra.run.smithi061.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-12-25T21:01:55.821 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-12-25T21:01:55.821 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-12-25T21:01:55.821 INFO:teuthology.orchestra.run.smithi061.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-12-25T21:01:55.821 INFO:teuthology.orchestra.run.smithi061.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-12-25T21:01:55.821 INFO:teuthology.orchestra.run.smithi061.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-12-25T21:01:55.821 INFO:teuthology.orchestra.run.smithi061.stdout: yajl-2.1.0-12.el8.x86_64 2023-12-25T21:01:55.821 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:01:55.821 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:01:56.108 DEBUG:teuthology.parallel:result is None 2023-12-25T21:01:56.108 INFO:teuthology.run_tasks:Running task pexec... 2023-12-25T21:01:56.119 INFO:teuthology.task.pexec:Executing custom commands... 2023-12-25T21:01:56.119 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi061.front.sepia.ceph.com 2023-12-25T21:01:56.120 DEBUG:teuthology.orchestra.run.smithi061:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-25T21:01:56.120 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi174.front.sepia.ceph.com 2023-12-25T21:01:56.120 DEBUG:teuthology.orchestra.run.smithi174:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-25T21:01:56.307 DEBUG:teuthology.parallel:result is None 2023-12-25T21:01:56.329 DEBUG:teuthology.parallel:result is None 2023-12-25T21:01:56.329 INFO:teuthology.run_tasks:Running task install... 2023-12-25T21:01:56.338 DEBUG:teuthology.task.install:project ceph 2023-12-25T21:01:56.338 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2'}} 2023-12-25T21:01:56.338 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2'} 2023-12-25T21:01:56.338 INFO:teuthology.task.install:Using flavor: default 2023-12-25T21:01:56.344 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-25T21:01:56.344 INFO:teuthology.task.install:extra packages: [] 2023-12-25T21:01:56.344 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-12-25T21:01:56.344 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-25T21:01:56.344 INFO:teuthology.packaging:ref: None 2023-12-25T21:01:56.345 INFO:teuthology.packaging:tag: v16.2.4 2023-12-25T21:01:56.345 INFO:teuthology.packaging:branch: None 2023-12-25T21:01:56.345 INFO:teuthology.packaging:sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T21:01:56.553 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-25T21:01:56.554 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-25T21:01:56.557 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-12-25T21:01:56.557 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-25T21:01:56.557 INFO:teuthology.packaging:ref: None 2023-12-25T21:01:56.557 INFO:teuthology.packaging:tag: v16.2.4 2023-12-25T21:01:56.557 INFO:teuthology.packaging:branch: None 2023-12-25T21:01:56.557 INFO:teuthology.packaging:sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T21:01:56.557 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-25T21:01:56.707 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-12-25T21:01:56.707 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-12-25T21:01:56.745 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-25T21:01:56.745 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-12-25T21:01:56.745 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-25T21:01:56.774 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-25T21:01:56.774 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-25T21:01:56.774 INFO:teuthology.packaging:ref: None 2023-12-25T21:01:56.774 INFO:teuthology.packaging:tag: v16.2.4 2023-12-25T21:01:56.774 INFO:teuthology.packaging:branch: None 2023-12-25T21:01:56.775 INFO:teuthology.packaging:sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T21:01:56.775 DEBUG:teuthology.orchestra.run.smithi061:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-25T21:01:56.781 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-12-25T21:01:56.781 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-12-25T21:01:56.805 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-25T21:01:56.806 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-12-25T21:01:56.806 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-25T21:01:56.838 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-25T21:01:56.838 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-25T21:01:56.838 INFO:teuthology.packaging:ref: None 2023-12-25T21:01:56.838 INFO:teuthology.packaging:tag: v16.2.4 2023-12-25T21:01:56.839 INFO:teuthology.packaging:branch: None 2023-12-25T21:01:56.839 INFO:teuthology.packaging:sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T21:01:56.839 DEBUG:teuthology.orchestra.run.smithi174:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-25T21:01:56.843 DEBUG:teuthology.orchestra.run.smithi061:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-25T21:01:56.910 DEBUG:teuthology.orchestra.run.smithi174:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-25T21:01:56.921 DEBUG:teuthology.orchestra.run.smithi061:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-25T21:01:56.989 INFO:teuthology.orchestra.run.smithi061.stdout:check_obsoletes = 1 2023-12-25T21:01:56.990 DEBUG:teuthology.orchestra.run.smithi174:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-25T21:01:56.991 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum clean all 2023-12-25T21:01:57.058 INFO:teuthology.orchestra.run.smithi174.stdout:check_obsoletes = 1 2023-12-25T21:01:57.059 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum clean all 2023-12-25T21:01:57.271 INFO:teuthology.orchestra.run.smithi061.stdout:76 files removed 2023-12-25T21:01:57.300 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-radosgw 2023-12-25T21:01:57.328 INFO:teuthology.orchestra.run.smithi174.stdout:76 files removed 2023-12-25T21:01:57.356 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-radosgw 2023-12-25T21:01:57.840 INFO:teuthology.orchestra.run.smithi061.stdout:ceph packages for x86_64 663 kB/s | 74 kB 00:00 2023-12-25T21:01:57.849 INFO:teuthology.orchestra.run.smithi174.stdout:ceph packages for x86_64 917 kB/s | 74 kB 00:00 2023-12-25T21:01:57.949 INFO:teuthology.orchestra.run.smithi061.stdout:ceph noarch packages 224 kB/s | 15 kB 00:00 2023-12-25T21:01:57.986 INFO:teuthology.orchestra.run.smithi174.stdout:ceph noarch packages 154 kB/s | 15 kB 00:00 2023-12-25T21:01:58.042 INFO:teuthology.orchestra.run.smithi061.stdout:ceph source packages 29 kB/s | 1.7 kB 00:00 2023-12-25T21:01:58.091 INFO:teuthology.orchestra.run.smithi174.stdout:ceph source packages 23 kB/s | 1.7 kB 00:00 2023-12-25T21:02:00.009 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 8 - AppStream 18 MB/s | 35 MB 00:01 2023-12-25T21:02:00.542 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 8 - AppStream 14 MB/s | 35 MB 00:02 2023-12-25T21:02:09.251 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 57 MB 00:02 2023-12-25T21:02:09.958 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 8 - BaseOS 19 MB/s | 57 MB 00:03 2023-12-25T21:02:18.109 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 8 - Extras 79 kB/s | 18 kB 00:00 2023-12-25T21:02:18.291 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 8 - Extras 85 kB/s | 18 kB 00:00 2023-12-25T21:02:18.468 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 8 - Extras common packages 42 kB/s | 6.9 kB 00:00 2023-12-25T21:02:18.814 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 6.9 kB 00:00 2023-12-25T21:02:19.770 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 8 - PowerTools 5.8 MB/s | 6.6 MB 00:01 2023-12-25T21:02:20.718 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 8 - PowerTools 3.8 MB/s | 6.6 MB 00:01 2023-12-25T21:02:21.271 INFO:teuthology.orchestra.run.smithi061.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-12-25T21:02:22.154 INFO:teuthology.orchestra.run.smithi174.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-12-25T21:02:23.473 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-12-25T21:02:23.749 INFO:teuthology.orchestra.run.smithi061.stdout:Extra Packages for Enterprise Linux 6.9 MB/s | 16 MB 00:02 2023-12-25T21:02:27.338 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 375 kB/s | 24 kB 00:00 2023-12-25T21:02:27.839 INFO:teuthology.orchestra.run.smithi061.stdout:lab-extras 419 kB/s | 24 kB 00:00 2023-12-25T21:02:32.024 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:02:32.029 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:02:32.029 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2023-12-25T21:02:32.029 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:02:32.030 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2023-12-25T21:02:32.030 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-12-25T21:02:32.030 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrading: 2023-12-25T21:02:32.030 INFO:teuthology.orchestra.run.smithi174.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-25T21:02:32.030 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-12-25T21:02:32.030 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2023-12-25T21:02:32.030 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-12-25T21:02:32.030 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-12-25T21:02:32.030 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-12-25T21:02:32.030 INFO:teuthology.orchestra.run.smithi174.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-12-25T21:02:32.030 INFO:teuthology.orchestra.run.smithi174.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-12-25T21:02:32.030 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-25T21:02:32.030 INFO:teuthology.orchestra.run.smithi174.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-25T21:02:32.031 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-12-25T21:02:32.031 INFO:teuthology.orchestra.run.smithi174.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-25T21:02:32.031 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-12-25T21:02:32.031 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-12-25T21:02:32.031 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-12-25T21:02:32.031 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2023-12-25T21:02:32.031 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-12-25T21:02:32.031 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-25T21:02:32.031 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-12-25T21:02:32.031 INFO:teuthology.orchestra.run.smithi174.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-12-25T21:02:32.031 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-12-25T21:02:32.031 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-12-25T21:02:32.031 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-12-25T21:02:32.032 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-12-25T21:02:32.032 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-12-25T21:02:32.032 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-12-25T21:02:32.032 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-12-25T21:02:32.032 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:02:32.032 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2023-12-25T21:02:32.032 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:02:32.032 INFO:teuthology.orchestra.run.smithi174.stdout:Install 25 Packages 2023-12-25T21:02:32.032 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrade 2 Packages 2023-12-25T21:02:32.032 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:02:32.032 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 59 M 2023-12-25T21:02:32.032 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2023-12-25T21:02:32.321 INFO:teuthology.orchestra.run.smithi174.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 34 MB/s | 6.0 MB 00:00 2023-12-25T21:02:32.347 INFO:teuthology.orchestra.run.smithi174.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 948 kB/s | 25 kB 00:00 2023-12-25T21:02:32.381 INFO:teuthology.orchestra.run.smithi174.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2023-12-25T21:02:32.415 INFO:teuthology.orchestra.run.smithi174.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-12-25T21:02:32.507 INFO:teuthology.orchestra.run.smithi174.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 44 MB/s | 4.1 MB 00:00 2023-12-25T21:02:32.524 INFO:teuthology.orchestra.run.smithi174.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 2.7 MB/s | 44 kB 00:00 2023-12-25T21:02:32.549 INFO:teuthology.orchestra.run.smithi174.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-12-25T21:02:32.591 INFO:teuthology.orchestra.run.smithi174.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 5.0 MB/s | 213 kB 00:00 2023-12-25T21:02:32.625 INFO:teuthology.orchestra.run.smithi174.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-12-25T21:02:32.683 INFO:teuthology.orchestra.run.smithi174.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 6.2 MB/s | 368 kB 00:00 2023-12-25T21:02:32.700 INFO:teuthology.orchestra.run.smithi174.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 6.8 MB/s | 114 kB 00:00 2023-12-25T21:02:32.775 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:02:32.778 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:02:32.778 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-12-25T21:02:32.778 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:02:32.778 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-12-25T21:02:32.778 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-12-25T21:02:32.778 INFO:teuthology.orchestra.run.smithi061.stdout:Upgrading: 2023-12-25T21:02:32.778 INFO:teuthology.orchestra.run.smithi061.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-25T21:02:32.778 INFO:teuthology.orchestra.run.smithi061.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-12-25T21:02:32.779 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-12-25T21:02:32.779 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-12-25T21:02:32.779 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-12-25T21:02:32.779 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-12-25T21:02:32.779 INFO:teuthology.orchestra.run.smithi061.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-12-25T21:02:32.779 INFO:teuthology.orchestra.run.smithi061.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-12-25T21:02:32.779 INFO:teuthology.orchestra.run.smithi061.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-25T21:02:32.779 INFO:teuthology.orchestra.run.smithi061.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-25T21:02:32.779 INFO:teuthology.orchestra.run.smithi061.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-12-25T21:02:32.779 INFO:teuthology.orchestra.run.smithi061.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-25T21:02:32.779 INFO:teuthology.orchestra.run.smithi061.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-12-25T21:02:32.779 INFO:teuthology.orchestra.run.smithi061.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-12-25T21:02:32.779 INFO:teuthology.orchestra.run.smithi061.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-12-25T21:02:32.779 INFO:teuthology.orchestra.run.smithi061.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2023-12-25T21:02:32.780 INFO:teuthology.orchestra.run.smithi061.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-12-25T21:02:32.780 INFO:teuthology.orchestra.run.smithi061.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-25T21:02:32.780 INFO:teuthology.orchestra.run.smithi061.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-12-25T21:02:32.780 INFO:teuthology.orchestra.run.smithi061.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-12-25T21:02:32.780 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-12-25T21:02:32.780 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-12-25T21:02:32.780 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-12-25T21:02:32.780 INFO:teuthology.orchestra.run.smithi061.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-12-25T21:02:32.780 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-12-25T21:02:32.780 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-12-25T21:02:32.780 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-12-25T21:02:32.780 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:02:32.780 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-12-25T21:02:32.780 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:02:32.781 INFO:teuthology.orchestra.run.smithi061.stdout:Install 25 Packages 2023-12-25T21:02:32.781 INFO:teuthology.orchestra.run.smithi061.stdout:Upgrade 2 Packages 2023-12-25T21:02:32.781 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:02:32.781 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 59 M 2023-12-25T21:02:32.781 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-12-25T21:02:32.842 INFO:teuthology.orchestra.run.smithi174.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2023-12-25T21:02:33.101 INFO:teuthology.orchestra.run.smithi174.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 25 MB/s | 24 MB 00:00 2023-12-25T21:02:33.184 INFO:teuthology.orchestra.run.smithi174.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 300 kB/s | 145 kB 00:00 2023-12-25T21:02:33.285 INFO:teuthology.orchestra.run.smithi174.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 437 kB/s | 44 kB 00:00 2023-12-25T21:02:33.452 INFO:teuthology.orchestra.run.smithi174.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 253 kB/s | 42 kB 00:00 2023-12-25T21:02:33.510 INFO:teuthology.orchestra.run.smithi174.stdout:(17/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 804 kB/s | 47 kB 00:00 2023-12-25T21:02:33.524 INFO:teuthology.orchestra.run.smithi061.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 32 MB/s | 6.0 MB 00:00 2023-12-25T21:02:33.549 INFO:teuthology.orchestra.run.smithi061.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 990 kB/s | 25 kB 00:00 2023-12-25T21:02:33.561 INFO:teuthology.orchestra.run.smithi174.stdout:(18/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 565 kB/s | 259 kB 00:00 2023-12-25T21:02:33.586 INFO:teuthology.orchestra.run.smithi174.stdout:(19/27): ndctl-libs-71.1-7.el8.x86_64.rpm 1.0 MB/s | 80 kB 00:00 2023-12-25T21:02:33.591 INFO:teuthology.orchestra.run.smithi061.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2023-12-25T21:02:33.625 INFO:teuthology.orchestra.run.smithi061.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-12-25T21:02:33.636 INFO:teuthology.orchestra.run.smithi174.stdout:(20/27): librdkafka-1.6.1-1.el8.x86_64.rpm 853 kB/s | 677 kB 00:00 2023-12-25T21:02:33.725 INFO:teuthology.orchestra.run.smithi061.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2023-12-25T21:02:33.750 INFO:teuthology.orchestra.run.smithi061.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-12-25T21:02:33.801 INFO:teuthology.orchestra.run.smithi061.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 1.5 MB/s | 80 kB 00:00 2023-12-25T21:02:33.834 INFO:teuthology.orchestra.run.smithi061.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 6.3 MB/s | 213 kB 00:00 2023-12-25T21:02:33.845 INFO:teuthology.orchestra.run.smithi174.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 410 kB/s | 116 kB 00:00 2023-12-25T21:02:33.868 INFO:teuthology.orchestra.run.smithi061.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-12-25T21:02:33.870 INFO:teuthology.orchestra.run.smithi174.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2023-12-25T21:02:33.901 INFO:teuthology.orchestra.run.smithi061.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2023-12-25T21:02:33.903 INFO:teuthology.orchestra.run.smithi174.stdout:(23/27): leveldb-1.22-1.el8.x86_64.rpm 701 kB/s | 181 kB 00:00 2023-12-25T21:02:33.927 INFO:teuthology.orchestra.run.smithi061.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-12-25T21:02:33.929 INFO:teuthology.orchestra.run.smithi174.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 703 kB/s | 59 kB 00:00 2023-12-25T21:02:33.954 INFO:teuthology.orchestra.run.smithi174.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 896 kB/s | 75 kB 00:00 2023-12-25T21:02:34.048 INFO:teuthology.orchestra.run.smithi174.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 26 MB/s | 3.7 MB 00:00 2023-12-25T21:02:34.052 INFO:teuthology.orchestra.run.smithi061.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2023-12-25T21:02:34.104 INFO:teuthology.orchestra.run.smithi174.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 23 MB/s | 4.0 MB 00:00 2023-12-25T21:02:34.110 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:02:34.110 INFO:teuthology.orchestra.run.smithi174.stdout:Total 28 MB/s | 59 MB 00:02 2023-12-25T21:02:34.178 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2023-12-25T21:02:34.227 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2023-12-25T21:02:34.227 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2023-12-25T21:02:34.319 INFO:teuthology.orchestra.run.smithi061.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:00 2023-12-25T21:02:34.427 INFO:teuthology.orchestra.run.smithi061.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 290 kB/s | 145 kB 00:00 2023-12-25T21:02:34.511 INFO:teuthology.orchestra.run.smithi061.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 527 kB/s | 44 kB 00:00 2023-12-25T21:02:34.586 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2023-12-25T21:02:34.590 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2023-12-25T21:02:34.728 INFO:teuthology.orchestra.run.smithi061.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 195 kB/s | 42 kB 00:00 2023-12-25T21:02:34.787 INFO:teuthology.orchestra.run.smithi061.stdout:(17/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 799 kB/s | 47 kB 00:00 2023-12-25T21:02:34.812 INFO:teuthology.orchestra.run.smithi061.stdout:(18/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 527 kB/s | 259 kB 00:00 2023-12-25T21:02:34.879 INFO:teuthology.orchestra.run.smithi061.stdout:(19/27): ndctl-libs-71.1-7.el8.x86_64.rpm 871 kB/s | 80 kB 00:00 2023-12-25T21:02:34.906 INFO:teuthology.orchestra.run.smithi061.stdout:(20/27): librdkafka-1.6.1-1.el8.x86_64.rpm 795 kB/s | 677 kB 00:00 2023-12-25T21:02:34.946 INFO:teuthology.orchestra.run.smithi061.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 869 kB/s | 116 kB 00:00 2023-12-25T21:02:34.988 INFO:teuthology.orchestra.run.smithi061.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 1.6 MB/s | 59 kB 00:00 2023-12-25T21:02:35.055 INFO:teuthology.orchestra.run.smithi061.stdout:(23/27): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2023-12-25T21:02:35.080 INFO:teuthology.orchestra.run.smithi061.stdout:(24/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.5 MB/s | 306 kB 00:00 2023-12-25T21:02:35.156 INFO:teuthology.orchestra.run.smithi061.stdout:(25/27): leveldb-1.22-1.el8.x86_64.rpm 735 kB/s | 181 kB 00:00 2023-12-25T21:02:35.256 INFO:teuthology.orchestra.run.smithi061.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 3.7 MB 00:00 2023-12-25T21:02:35.342 INFO:teuthology.orchestra.run.smithi061.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 15 MB/s | 4.0 MB 00:00 2023-12-25T21:02:35.347 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:02:35.348 INFO:teuthology.orchestra.run.smithi061.stdout:Total 23 MB/s | 59 MB 00:02 2023-12-25T21:02:35.419 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-12-25T21:02:35.472 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-12-25T21:02:35.472 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-12-25T21:02:35.749 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2023-12-25T21:02:35.839 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-12-25T21:02:35.842 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-12-25T21:02:35.983 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-12-25T21:02:36.195 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-12-25T21:02:36.403 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-25T21:02:36.813 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-25T21:02:37.042 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-12-25T21:02:37.070 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-25T21:02:37.164 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-25T21:02:37.280 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-12-25T21:02:37.436 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-12-25T21:02:37.501 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-12-25T21:02:37.631 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-25T21:02:37.683 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-25T21:02:37.693 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-25T21:02:37.904 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-12-25T21:02:38.127 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-25T21:02:38.335 INFO:teuthology.orchestra.run.smithi061.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-25T21:02:38.459 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-25T21:02:38.481 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2023-12-25T21:02:38.653 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-25T21:02:38.718 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-12-25T21:02:38.732 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-25T21:02:38.883 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-12-25T21:02:38.893 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-25T21:02:38.944 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-25T21:02:39.078 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-25T21:02:39.146 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-25T21:02:39.216 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-12-25T21:02:39.316 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-12-25T21:02:39.503 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-12-25T21:02:39.686 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-25T21:02:39.756 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-25T21:02:39.803 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2023-12-25T21:02:39.936 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-25T21:02:40.001 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-25T21:02:40.075 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-25T21:02:40.355 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-25T21:02:40.474 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-12-25T21:02:40.553 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-25T21:02:40.650 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-25T21:02:40.685 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-25T21:02:40.751 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-25T21:02:40.865 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-12-25T21:02:40.967 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-12-25T21:02:41.071 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-12-25T21:02:41.137 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-12-25T21:02:41.252 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-25T21:02:41.324 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-25T21:02:41.335 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-25T21:02:41.411 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-25T21:02:41.503 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-12-25T21:02:41.601 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-25T21:02:41.711 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-12-25T21:02:41.910 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-12-25T21:02:42.067 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-25T21:02:42.145 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-12-25T21:02:42.301 INFO:teuthology.orchestra.run.smithi061.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-25T21:02:42.410 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-25T21:02:42.622 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-12-25T21:02:42.890 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-12-25T21:02:43.034 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-25T21:02:43.124 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-25T21:02:43.313 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-12-25T21:02:43.527 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-12-25T21:02:43.747 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-12-25T21:02:44.002 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-12-25T21:02:44.684 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-25T21:02:44.911 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-25T21:02:45.606 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-25T21:02:45.892 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-25T21:02:45.985 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-25T21:02:46.143 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-25T21:02:46.729 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-25T21:02:46.976 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-25T21:02:47.770 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-25T21:02:48.075 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-25T21:02:48.181 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-25T21:02:48.350 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-25T21:03:04.353 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-25T21:03:04.354 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /sys 2023-12-25T21:03:04.354 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /proc 2023-12-25T21:03:04.354 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /dev 2023-12-25T21:03:04.354 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /run 2023-12-25T21:03:04.354 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /mnt 2023-12-25T21:03:04.354 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /var/tmp 2023-12-25T21:03:04.354 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /home 2023-12-25T21:03:04.354 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /tmp 2023-12-25T21:03:04.354 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /dev 2023-12-25T21:03:04.354 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:05.681 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-25T21:03:05.761 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-25T21:03:05.761 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-25T21:03:05.977 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-25T21:03:05.977 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-25T21:03:06.726 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-25T21:03:06.726 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-12-25T21:03:06.726 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-12-25T21:03:06.726 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-12-25T21:03:06.726 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-12-25T21:03:06.726 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-12-25T21:03:06.726 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-12-25T21:03:06.726 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-12-25T21:03:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-12-25T21:03:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-12-25T21:03:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-12-25T21:03:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-12-25T21:03:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-12-25T21:03:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-12-25T21:03:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-12-25T21:03:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2023-12-25T21:03:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-12-25T21:03:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-12-25T21:03:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-12-25T21:03:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-12-25T21:03:06.727 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-12-25T21:03:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-12-25T21:03:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-12-25T21:03:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-12-25T21:03:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-12-25T21:03:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-12-25T21:03:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-12-25T21:03:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-12-25T21:03:06.728 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-12-25T21:03:07.359 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-25T21:03:07.359 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /sys 2023-12-25T21:03:07.359 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /proc 2023-12-25T21:03:07.359 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /dev 2023-12-25T21:03:07.359 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /run 2023-12-25T21:03:07.359 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /mnt 2023-12-25T21:03:07.359 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /var/tmp 2023-12-25T21:03:07.360 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /home 2023-12-25T21:03:07.360 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /tmp 2023-12-25T21:03:07.360 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /dev 2023-12-25T21:03:07.360 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:03:07.989 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-12-25T21:03:07.990 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:07.990 INFO:teuthology.orchestra.run.smithi174.stdout:Upgraded: 2023-12-25T21:03:07.990 INFO:teuthology.orchestra.run.smithi174.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:07.990 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2023-12-25T21:03:07.990 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:07.991 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:07.991 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:07.991 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:07.991 INFO:teuthology.orchestra.run.smithi174.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-12-25T21:03:07.991 INFO:teuthology.orchestra.run.smithi174.stdout: fmt-6.2.1-1.el8.x86_64 2023-12-25T21:03:07.991 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-25T21:03:07.992 INFO:teuthology.orchestra.run.smithi174.stdout: leveldb-1.22-1.el8.x86_64 2023-12-25T21:03:07.992 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:07.992 INFO:teuthology.orchestra.run.smithi174.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-25T21:03:07.992 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-12-25T21:03:07.992 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-25T21:03:07.992 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:07.993 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka-1.6.1-1.el8.x86_64 2023-12-25T21:03:07.993 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:07.993 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-25T21:03:07.993 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-25T21:03:07.993 INFO:teuthology.orchestra.run.smithi174.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-12-25T21:03:07.993 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:07.994 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:07.994 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:07.994 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-25T21:03:07.994 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:07.994 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:07.994 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:07.994 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:07.994 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:03:08.271 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-test 2023-12-25T21:03:08.764 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-25T21:03:08.866 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-25T21:03:08.866 INFO:teuthology.orchestra.run.smithi061.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-25T21:03:09.092 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-25T21:03:09.092 INFO:teuthology.orchestra.run.smithi061.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-25T21:03:09.128 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:42 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:03:09.899 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-25T21:03:09.899 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-12-25T21:03:09.899 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-12-25T21:03:09.899 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-12-25T21:03:09.899 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-12-25T21:03:09.900 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-12-25T21:03:09.900 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-12-25T21:03:09.900 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-12-25T21:03:09.900 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-12-25T21:03:09.900 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-12-25T21:03:09.901 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-12-25T21:03:09.901 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-12-25T21:03:09.901 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-12-25T21:03:09.901 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-12-25T21:03:09.901 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-12-25T21:03:09.901 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2023-12-25T21:03:09.902 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-12-25T21:03:09.902 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-12-25T21:03:09.902 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-12-25T21:03:09.902 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-12-25T21:03:09.902 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-12-25T21:03:09.902 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-12-25T21:03:09.902 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-12-25T21:03:09.902 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-12-25T21:03:09.903 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-12-25T21:03:09.903 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-12-25T21:03:09.903 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-12-25T21:03:09.903 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-12-25T21:03:09.903 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-12-25T21:03:10.824 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:03:10.825 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:03:10.825 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2023-12-25T21:03:10.825 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:03:10.825 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2023-12-25T21:03:10.825 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-12-25T21:03:10.825 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2023-12-25T21:03:10.825 INFO:teuthology.orchestra.run.smithi174.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-12-25T21:03:10.825 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-12-25T21:03:10.825 INFO:teuthology.orchestra.run.smithi174.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-12-25T21:03:10.825 INFO:teuthology.orchestra.run.smithi174.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-12-25T21:03:10.825 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-12-25T21:03:10.826 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:10.826 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2023-12-25T21:03:10.826 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:03:10.826 INFO:teuthology.orchestra.run.smithi174.stdout:Install 6 Packages 2023-12-25T21:03:10.826 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:10.826 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 53 M 2023-12-25T21:03:10.826 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 229 M 2023-12-25T21:03:10.827 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2023-12-25T21:03:11.076 INFO:teuthology.orchestra.run.smithi174.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.6 MB/s | 169 kB 00:00 2023-12-25T21:03:11.253 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-12-25T21:03:11.253 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:03:11.253 INFO:teuthology.orchestra.run.smithi061.stdout:Upgraded: 2023-12-25T21:03:11.253 INFO:teuthology.orchestra.run.smithi061.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:11.253 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-12-25T21:03:11.253 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:11.253 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:11.253 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:11.254 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:11.254 INFO:teuthology.orchestra.run.smithi061.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-12-25T21:03:11.254 INFO:teuthology.orchestra.run.smithi061.stdout: fmt-6.2.1-1.el8.x86_64 2023-12-25T21:03:11.254 INFO:teuthology.orchestra.run.smithi061.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-25T21:03:11.254 INFO:teuthology.orchestra.run.smithi061.stdout: leveldb-1.22-1.el8.x86_64 2023-12-25T21:03:11.254 INFO:teuthology.orchestra.run.smithi061.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:11.254 INFO:teuthology.orchestra.run.smithi061.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-25T21:03:11.254 INFO:teuthology.orchestra.run.smithi061.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-12-25T21:03:11.254 INFO:teuthology.orchestra.run.smithi061.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-25T21:03:11.254 INFO:teuthology.orchestra.run.smithi061.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:11.254 INFO:teuthology.orchestra.run.smithi061.stdout: librdkafka-1.6.1-1.el8.x86_64 2023-12-25T21:03:11.254 INFO:teuthology.orchestra.run.smithi061.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:11.254 INFO:teuthology.orchestra.run.smithi061.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-25T21:03:11.254 INFO:teuthology.orchestra.run.smithi061.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-25T21:03:11.255 INFO:teuthology.orchestra.run.smithi061.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-12-25T21:03:11.255 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:11.255 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:11.255 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:11.255 INFO:teuthology.orchestra.run.smithi061.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-25T21:03:11.255 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:11.255 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:11.255 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:11.255 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:03:11.255 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:03:11.518 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-test 2023-12-25T21:03:11.718 INFO:teuthology.orchestra.run.smithi174.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 74 MB/s | 52 MB 00:00 2023-12-25T21:03:11.768 INFO:teuthology.orchestra.run.smithi174.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 269 kB/s | 203 kB 00:00 2023-12-25T21:03:11.793 INFO:teuthology.orchestra.run.smithi174.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 261 kB/s | 187 kB 00:00 2023-12-25T21:03:11.885 INFO:teuthology.orchestra.run.smithi174.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 772 kB/s | 71 kB 00:00 2023-12-25T21:03:12.201 INFO:teuthology.orchestra.run.smithi174.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 669 kB/s | 323 kB 00:00 2023-12-25T21:03:12.202 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:03:12.203 INFO:teuthology.orchestra.run.smithi174.stdout:Total 38 MB/s | 53 MB 00:01 2023-12-25T21:03:12.246 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2023-12-25T21:03:12.267 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2023-12-25T21:03:12.267 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2023-12-25T21:03:12.412 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:00:45 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:03:12.574 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2023-12-25T21:03:12.577 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2023-12-25T21:03:13.077 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2023-12-25T21:03:13.267 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-12-25T21:03:13.450 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-12-25T21:03:13.649 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-25T21:03:13.715 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-25T21:03:13.880 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-12-25T21:03:14.066 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-25T21:03:14.226 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:03:14.227 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:03:14.227 INFO:teuthology.orchestra.run.smithi061.stdout: Package Architecture Version Repository Size 2023-12-25T21:03:14.227 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:03:14.227 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-12-25T21:03:14.227 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-12-25T21:03:14.227 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-12-25T21:03:14.227 INFO:teuthology.orchestra.run.smithi061.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-12-25T21:03:14.227 INFO:teuthology.orchestra.run.smithi061.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-12-25T21:03:14.227 INFO:teuthology.orchestra.run.smithi061.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-12-25T21:03:14.227 INFO:teuthology.orchestra.run.smithi061.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-12-25T21:03:14.227 INFO:teuthology.orchestra.run.smithi061.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-12-25T21:03:14.227 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:03:14.227 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-12-25T21:03:14.228 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:03:14.228 INFO:teuthology.orchestra.run.smithi061.stdout:Install 6 Packages 2023-12-25T21:03:14.228 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:03:14.228 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 53 M 2023-12-25T21:03:14.228 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 229 M 2023-12-25T21:03:14.228 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-12-25T21:03:14.481 INFO:teuthology.orchestra.run.smithi061.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.1 MB/s | 169 kB 00:00 2023-12-25T21:03:15.181 INFO:teuthology.orchestra.run.smithi061.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 69 MB/s | 52 MB 00:00 2023-12-25T21:03:15.206 INFO:teuthology.orchestra.run.smithi061.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 261 kB/s | 203 kB 00:00 2023-12-25T21:03:15.231 INFO:teuthology.orchestra.run.smithi061.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 250 kB/s | 187 kB 00:00 2023-12-25T21:03:15.332 INFO:teuthology.orchestra.run.smithi061.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 705 kB/s | 71 kB 00:00 2023-12-25T21:03:15.632 INFO:teuthology.orchestra.run.smithi061.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 717 kB/s | 323 kB 00:00 2023-12-25T21:03:15.634 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:03:15.634 INFO:teuthology.orchestra.run.smithi061.stdout:Total 38 MB/s | 53 MB 00:01 2023-12-25T21:03:15.678 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-12-25T21:03:15.701 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-12-25T21:03:15.701 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-12-25T21:03:16.028 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-12-25T21:03:16.031 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-12-25T21:03:16.550 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-12-25T21:03:16.761 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-12-25T21:03:16.976 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-12-25T21:03:17.205 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-25T21:03:17.279 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-25T21:03:17.489 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-12-25T21:03:17.647 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-25T21:03:18.912 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-25T21:03:19.145 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-25T21:03:19.455 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-25T21:03:19.457 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-12-25T21:03:19.457 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-12-25T21:03:19.457 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-12-25T21:03:19.458 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-25T21:03:19.458 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-25T21:03:20.211 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-12-25T21:03:20.211 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:20.211 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2023-12-25T21:03:20.211 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-12-25T21:03:20.211 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-12-25T21:03:20.211 INFO:teuthology.orchestra.run.smithi174.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-12-25T21:03:20.212 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:20.212 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:03:20.448 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph 2023-12-25T21:03:21.310 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:54 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:03:22.811 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-25T21:03:22.995 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:03:22.998 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:03:22.998 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2023-12-25T21:03:22.999 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:03:22.999 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2023-12-25T21:03:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-12-25T21:03:22.999 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2023-12-25T21:03:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-12-25T21:03:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-12-25T21:03:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-12-25T21:03:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-12-25T21:03:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-12-25T21:03:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-25T21:03:22.999 INFO:teuthology.orchestra.run.smithi174.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-25T21:03:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-12-25T21:03:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-25T21:03:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-25T21:03:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-12-25T21:03:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-25T21:03:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-25T21:03:23.000 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-25T21:03:23.001 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-12-25T21:03:23.001 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-12-25T21:03:23.001 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-25T21:03:23.001 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-25T21:03:23.001 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-25T21:03:23.001 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-12-25T21:03:23.002 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-12-25T21:03:23.002 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-12-25T21:03:23.002 INFO:teuthology.orchestra.run.smithi174.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-25T21:03:23.002 INFO:teuthology.orchestra.run.smithi174.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-25T21:03:23.002 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-25T21:03:23.002 INFO:teuthology.orchestra.run.smithi174.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-25T21:03:23.003 INFO:teuthology.orchestra.run.smithi174.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-25T21:03:23.003 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-25T21:03:23.003 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-25T21:03:23.003 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-12-25T21:03:23.003 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-25T21:03:23.003 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:23.003 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2023-12-25T21:03:23.003 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:03:23.003 INFO:teuthology.orchestra.run.smithi174.stdout:Install 32 Packages 2023-12-25T21:03:23.004 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:23.004 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 31 M 2023-12-25T21:03:23.005 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 111 M 2023-12-25T21:03:23.005 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2023-12-25T21:03:23.070 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-25T21:03:23.374 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-25T21:03:23.374 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-12-25T21:03:23.375 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-12-25T21:03:23.375 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-12-25T21:03:23.375 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-25T21:03:23.375 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-25T21:03:23.769 INFO:teuthology.orchestra.run.smithi174.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 148 kB/s | 6.3 kB 00:00 2023-12-25T21:03:23.819 INFO:teuthology.orchestra.run.smithi174.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2023-12-25T21:03:23.869 INFO:teuthology.orchestra.run.smithi174.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 17 MB/s | 2.4 MB 00:00 2023-12-25T21:03:23.912 INFO:teuthology.orchestra.run.smithi174.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 5.2 MB/s | 220 kB 00:00 2023-12-25T21:03:24.004 INFO:teuthology.orchestra.run.smithi174.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2023-12-25T21:03:24.152 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-12-25T21:03:24.152 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:03:24.153 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-12-25T21:03:24.153 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-12-25T21:03:24.153 INFO:teuthology.orchestra.run.smithi061.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-12-25T21:03:24.153 INFO:teuthology.orchestra.run.smithi061.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-12-25T21:03:24.153 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:03:24.153 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:03:24.237 INFO:teuthology.orchestra.run.smithi174.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 44 MB/s | 18 MB 00:00 2023-12-25T21:03:24.337 INFO:teuthology.orchestra.run.smithi174.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 369 kB/s | 157 kB 00:00 2023-12-25T21:03:24.373 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph 2023-12-25T21:03:24.422 INFO:teuthology.orchestra.run.smithi174.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 93 kB/s | 39 kB 00:00 2023-12-25T21:03:24.454 INFO:teuthology.orchestra.run.smithi174.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 488 kB/s | 57 kB 00:00 2023-12-25T21:03:24.638 INFO:teuthology.orchestra.run.smithi174.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 258 kB/s | 103 kB 00:00 2023-12-25T21:03:24.697 INFO:teuthology.orchestra.run.smithi174.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.0 MB/s | 249 kB 00:00 2023-12-25T21:03:24.739 INFO:teuthology.orchestra.run.smithi174.stdout:(12/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.4 MB/s | 457 kB 00:00 2023-12-25T21:03:24.764 INFO:teuthology.orchestra.run.smithi174.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.6 MB/s | 109 kB 00:00 2023-12-25T21:03:24.889 INFO:teuthology.orchestra.run.smithi174.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 2.5 MB/s | 641 kB 00:00 2023-12-25T21:03:25.198 INFO:teuthology.orchestra.run.smithi174.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 95 kB/s | 44 kB 00:00 2023-12-25T21:03:25.260 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:00:58 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:03:25.332 INFO:teuthology.orchestra.run.smithi174.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 327 kB/s | 185 kB 00:00 2023-12-25T21:03:25.407 INFO:teuthology.orchestra.run.smithi174.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 334 kB/s | 173 kB 00:00 2023-12-25T21:03:25.490 INFO:teuthology.orchestra.run.smithi174.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 70 kB/s | 11 kB 00:00 2023-12-25T21:03:25.541 INFO:teuthology.orchestra.run.smithi174.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 131 kB/s | 18 kB 00:00 2023-12-25T21:03:25.616 INFO:teuthology.orchestra.run.smithi174.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 921 kB/s | 384 kB 00:00 2023-12-25T21:03:25.632 INFO:teuthology.orchestra.run.smithi174.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 343 kB/s | 49 kB 00:00 2023-12-25T21:03:25.683 INFO:teuthology.orchestra.run.smithi174.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 413 kB/s | 59 kB 00:00 2023-12-25T21:03:25.792 INFO:teuthology.orchestra.run.smithi174.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 98 kB/s | 16 kB 00:00 2023-12-25T21:03:25.817 INFO:teuthology.orchestra.run.smithi174.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2023-12-25T21:03:25.842 INFO:teuthology.orchestra.run.smithi174.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 118 kB/s | 19 kB 00:00 2023-12-25T21:03:25.934 INFO:teuthology.orchestra.run.smithi174.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 168 kB/s | 24 kB 00:00 2023-12-25T21:03:25.968 INFO:teuthology.orchestra.run.smithi174.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 194 kB/s | 29 kB 00:00 2023-12-25T21:03:25.993 INFO:teuthology.orchestra.run.smithi174.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 183 kB/s | 27 kB 00:00 2023-12-25T21:03:26.168 INFO:teuthology.orchestra.run.smithi174.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 488 kB/s | 85 kB 00:00 2023-12-25T21:03:26.193 INFO:teuthology.orchestra.run.smithi174.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.1 MB/s | 251 kB 00:00 2023-12-25T21:03:26.218 INFO:teuthology.orchestra.run.smithi174.stdout:(31/32): python3-waitress-1.4.3-1.el8.noarch.rp 899 kB/s | 255 kB 00:00 2023-12-25T21:03:26.293 INFO:teuthology.orchestra.run.smithi174.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 183 kB/s | 23 kB 00:00 2023-12-25T21:03:26.295 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:03:26.295 INFO:teuthology.orchestra.run.smithi174.stdout:Total 9.4 MB/s | 31 MB 00:03 2023-12-25T21:03:26.377 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2023-12-25T21:03:26.415 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2023-12-25T21:03:26.415 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2023-12-25T21:03:26.735 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2023-12-25T21:03:26.738 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2023-12-25T21:03:27.057 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:03:27.061 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:03:27.061 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-12-25T21:03:27.061 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:03:27.061 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-12-25T21:03:27.061 INFO:teuthology.orchestra.run.smithi061.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-12-25T21:03:27.061 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-12-25T21:03:27.061 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-12-25T21:03:27.061 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-12-25T21:03:27.061 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-12-25T21:03:27.061 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-12-25T21:03:27.062 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-12-25T21:03:27.062 INFO:teuthology.orchestra.run.smithi061.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-25T21:03:27.062 INFO:teuthology.orchestra.run.smithi061.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-25T21:03:27.062 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-12-25T21:03:27.062 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-25T21:03:27.062 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-25T21:03:27.062 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-12-25T21:03:27.062 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-25T21:03:27.062 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-25T21:03:27.062 INFO:teuthology.orchestra.run.smithi061.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-25T21:03:27.062 INFO:teuthology.orchestra.run.smithi061.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-12-25T21:03:27.062 INFO:teuthology.orchestra.run.smithi061.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-12-25T21:03:27.062 INFO:teuthology.orchestra.run.smithi061.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-25T21:03:27.063 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-25T21:03:27.063 INFO:teuthology.orchestra.run.smithi061.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-25T21:03:27.063 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-12-25T21:03:27.063 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-12-25T21:03:27.063 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-12-25T21:03:27.063 INFO:teuthology.orchestra.run.smithi061.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-25T21:03:27.063 INFO:teuthology.orchestra.run.smithi061.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-25T21:03:27.063 INFO:teuthology.orchestra.run.smithi061.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-25T21:03:27.063 INFO:teuthology.orchestra.run.smithi061.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-25T21:03:27.063 INFO:teuthology.orchestra.run.smithi061.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-25T21:03:27.063 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-25T21:03:27.063 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-25T21:03:27.063 INFO:teuthology.orchestra.run.smithi061.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-12-25T21:03:27.063 INFO:teuthology.orchestra.run.smithi061.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-25T21:03:27.064 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:03:27.064 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-12-25T21:03:27.064 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:03:27.064 INFO:teuthology.orchestra.run.smithi061.stdout:Install 32 Packages 2023-12-25T21:03:27.064 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:03:27.064 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 31 M 2023-12-25T21:03:27.064 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 111 M 2023-12-25T21:03:27.065 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-12-25T21:03:27.409 INFO:teuthology.orchestra.run.smithi061.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 123 kB/s | 6.3 kB 00:00 2023-12-25T21:03:27.459 INFO:teuthology.orchestra.run.smithi061.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2023-12-25T21:03:27.518 INFO:teuthology.orchestra.run.smithi061.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2023-12-25T21:03:27.543 INFO:teuthology.orchestra.run.smithi061.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2023-12-25T21:03:27.635 INFO:teuthology.orchestra.run.smithi061.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 22 MB/s | 5.0 MB 00:00 2023-12-25T21:03:27.885 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2023-12-25T21:03:27.910 INFO:teuthology.orchestra.run.smithi061.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 40 MB/s | 18 MB 00:00 2023-12-25T21:03:28.002 INFO:teuthology.orchestra.run.smithi061.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 342 kB/s | 157 kB 00:00 2023-12-25T21:03:28.069 INFO:teuthology.orchestra.run.smithi061.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 89 kB/s | 39 kB 00:00 2023-12-25T21:03:28.084 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-25T21:03:28.086 INFO:teuthology.orchestra.run.smithi061.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 683 kB/s | 57 kB 00:00 2023-12-25T21:03:28.303 INFO:teuthology.orchestra.run.smithi061.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.1 MB/s | 249 kB 00:00 2023-12-25T21:03:28.311 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-25T21:03:28.337 INFO:teuthology.orchestra.run.smithi061.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 243 kB/s | 103 kB 00:00 2023-12-25T21:03:28.370 INFO:teuthology.orchestra.run.smithi061.stdout:(12/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.5 MB/s | 457 kB 00:00 2023-12-25T21:03:28.404 INFO:teuthology.orchestra.run.smithi061.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 6.2 MB/s | 641 kB 00:00 2023-12-25T21:03:28.521 INFO:teuthology.orchestra.run.smithi061.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 593 kB/s | 109 kB 00:00 2023-12-25T21:03:28.578 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-12-25T21:03:28.818 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-12-25T21:03:28.888 INFO:teuthology.orchestra.run.smithi061.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 84 kB/s | 44 kB 00:00 2023-12-25T21:03:28.975 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-12-25T21:03:29.005 INFO:teuthology.orchestra.run.smithi061.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 308 kB/s | 185 kB 00:00 2023-12-25T21:03:29.130 INFO:teuthology.orchestra.run.smithi061.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 89 kB/s | 11 kB 00:00 2023-12-25T21:03:29.138 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-12-25T21:03:29.239 INFO:teuthology.orchestra.run.smithi061.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 241 kB/s | 173 kB 00:00 2023-12-25T21:03:29.264 INFO:teuthology.orchestra.run.smithi061.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 132 kB/s | 18 kB 00:00 2023-12-25T21:03:29.340 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-12-25T21:03:29.348 INFO:teuthology.orchestra.run.smithi061.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 449 kB/s | 49 kB 00:00 2023-12-25T21:03:29.373 INFO:teuthology.orchestra.run.smithi061.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 540 kB/s | 59 kB 00:00 2023-12-25T21:03:29.481 INFO:teuthology.orchestra.run.smithi061.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 648 kB/s | 384 kB 00:00 2023-12-25T21:03:29.493 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-12-25T21:03:29.507 INFO:teuthology.orchestra.run.smithi061.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 117 kB/s | 16 kB 00:00 2023-12-25T21:03:29.582 INFO:teuthology.orchestra.run.smithi061.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 186 kB/s | 19 kB 00:00 2023-12-25T21:03:29.607 INFO:teuthology.orchestra.run.smithi061.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 239 kB/s | 24 kB 00:00 2023-12-25T21:03:29.683 INFO:teuthology.orchestra.run.smithi061.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 292 kB/s | 29 kB 00:00 2023-12-25T21:03:29.733 INFO:teuthology.orchestra.run.smithi061.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 220 kB/s | 27 kB 00:00 2023-12-25T21:03:29.735 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-12-25T21:03:29.775 INFO:teuthology.orchestra.run.smithi061.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 663 kB/s | 283 kB 00:00 2023-12-25T21:03:29.958 INFO:teuthology.orchestra.run.smithi061.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 466 kB/s | 85 kB 00:00 2023-12-25T21:03:29.963 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-12-25T21:03:30.042 INFO:teuthology.orchestra.run.smithi061.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 711 kB/s | 255 kB 00:00 2023-12-25T21:03:30.059 INFO:teuthology.orchestra.run.smithi061.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2023-12-25T21:03:30.092 INFO:teuthology.orchestra.run.smithi061.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 699 kB/s | 251 kB 00:00 2023-12-25T21:03:30.096 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:03:30.097 INFO:teuthology.orchestra.run.smithi061.stdout:Total 10 MB/s | 31 MB 00:03 2023-12-25T21:03:30.181 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-12-25T21:03:30.223 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-12-25T21:03:30.223 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-12-25T21:03:30.232 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-12-25T21:03:30.484 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-12-25T21:03:30.537 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-12-25T21:03:30.541 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-12-25T21:03:30.858 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-12-25T21:03:31.131 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-12-25T21:03:31.303 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-12-25T21:03:31.564 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-12-25T21:03:31.724 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-12-25T21:03:31.729 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-25T21:03:31.962 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-25T21:03:32.038 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-12-25T21:03:32.202 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-25T21:03:32.250 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-12-25T21:03:32.457 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-12-25T21:03:32.464 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-12-25T21:03:32.660 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-12-25T21:03:32.664 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-25T21:03:32.883 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-12-25T21:03:32.933 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-25T21:03:33.064 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-12-25T21:03:33.158 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-25T21:03:33.248 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-12-25T21:03:33.420 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-12-25T21:03:33.426 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-12-25T21:03:33.630 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-12-25T21:03:33.751 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-12-25T21:03:33.874 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-12-25T21:03:34.026 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-25T21:03:34.159 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-12-25T21:03:34.421 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-12-25T21:03:34.422 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-12-25T21:03:34.651 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-25T21:03:34.842 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-12-25T21:03:35.124 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-12-25T21:03:35.336 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-12-25T21:03:35.574 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-12-25T21:03:35.771 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-25T21:03:36.086 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-12-25T21:03:36.185 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-25T21:03:36.310 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-12-25T21:03:36.418 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-25T21:03:36.548 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-12-25T21:03:36.780 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-25T21:03:36.851 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-25T21:03:37.072 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-25T21:03:37.076 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-25T21:03:37.299 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-25T21:03:37.316 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-25T21:03:37.461 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-25T21:03:37.479 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-25T21:03:37.578 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-12-25T21:03:37.635 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-25T21:03:37.959 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-12-25T21:03:38.045 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-25T21:03:38.045 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-12-25T21:03:38.045 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-12-25T21:03:38.045 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-12-25T21:03:38.045 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-12-25T21:03:38.046 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-12-25T21:03:38.046 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-12-25T21:03:38.046 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-12-25T21:03:38.046 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-12-25T21:03:38.046 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-12-25T21:03:38.046 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-12-25T21:03:38.046 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-12-25T21:03:38.046 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-12-25T21:03:38.046 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-12-25T21:03:38.046 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-12-25T21:03:38.046 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-12-25T21:03:38.046 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-12-25T21:03:38.046 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-25T21:03:38.047 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-12-25T21:03:38.047 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-12-25T21:03:38.047 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-12-25T21:03:38.047 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-12-25T21:03:38.047 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-12-25T21:03:38.047 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-12-25T21:03:38.047 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-12-25T21:03:38.047 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-12-25T21:03:38.047 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-12-25T21:03:38.047 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-12-25T21:03:38.047 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-12-25T21:03:38.047 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-12-25T21:03:38.047 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-12-25T21:03:38.047 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-12-25T21:03:38.290 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-25T21:03:38.690 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-12-25T21:03:38.902 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-25T21:03:39.164 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-12-25T21:03:39.164 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:39.164 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2023-12-25T21:03:39.164 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:39.164 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:39.164 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:39.164 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-12-25T21:03:39.165 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:39.165 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:39.165 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-25T21:03:39.165 INFO:teuthology.orchestra.run.smithi174.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-25T21:03:39.165 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-25T21:03:39.165 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-25T21:03:39.165 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-25T21:03:39.165 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-25T21:03:39.165 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-25T21:03:39.165 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-25T21:03:39.165 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-25T21:03:39.165 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-25T21:03:39.165 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-25T21:03:39.166 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-25T21:03:39.166 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-25T21:03:39.166 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend-2.6-1.el8.noarch 2023-12-25T21:03:39.166 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-25T21:03:39.166 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-25T21:03:39.166 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-25T21:03:39.166 INFO:teuthology.orchestra.run.smithi174.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-25T21:03:39.166 INFO:teuthology.orchestra.run.smithi174.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-25T21:03:39.166 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-25T21:03:39.166 INFO:teuthology.orchestra.run.smithi174.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-25T21:03:39.166 INFO:teuthology.orchestra.run.smithi174.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-25T21:03:39.166 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-25T21:03:39.166 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-25T21:03:39.166 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-25T21:03:39.167 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-25T21:03:39.167 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:39.167 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:03:39.945 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-base 2023-12-25T21:03:40.531 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-25T21:03:40.735 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-25T21:03:40.806 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:13 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:03:41.194 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-25T21:03:41.443 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-25T21:03:41.678 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-25T21:03:42.267 INFO:teuthology.orchestra.run.smithi174.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T21:03:42.378 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-25T21:03:42.400 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-25T21:03:42.495 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:03:42.496 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2023-12-25T21:03:42.496 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:03:42.570 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-25T21:03:42.640 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install cephadm 2023-12-25T21:03:43.002 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-25T21:03:43.002 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-12-25T21:03:43.002 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-12-25T21:03:43.003 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-12-25T21:03:43.003 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-12-25T21:03:43.003 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-12-25T21:03:43.003 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-12-25T21:03:43.003 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-12-25T21:03:43.003 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-12-25T21:03:43.003 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-12-25T21:03:43.003 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-12-25T21:03:43.003 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-12-25T21:03:43.003 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-12-25T21:03:43.003 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-12-25T21:03:43.003 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-12-25T21:03:43.003 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-12-25T21:03:43.003 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-12-25T21:03:43.004 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-25T21:03:43.004 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-12-25T21:03:43.004 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-12-25T21:03:43.004 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-12-25T21:03:43.004 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-12-25T21:03:43.004 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-12-25T21:03:43.004 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-12-25T21:03:43.004 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-12-25T21:03:43.004 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-12-25T21:03:43.005 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-12-25T21:03:43.005 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-12-25T21:03:43.005 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-12-25T21:03:43.005 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-12-25T21:03:43.005 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-12-25T21:03:43.005 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-12-25T21:03:43.493 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:16 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:03:44.249 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-12-25T21:03:44.250 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:03:44.250 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-12-25T21:03:44.250 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:44.250 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:44.250 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:44.250 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-12-25T21:03:44.250 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:44.250 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:44.250 INFO:teuthology.orchestra.run.smithi061.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-25T21:03:44.250 INFO:teuthology.orchestra.run.smithi061.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-25T21:03:44.250 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-25T21:03:44.250 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-25T21:03:44.250 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-25T21:03:44.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-25T21:03:44.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-25T21:03:44.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-25T21:03:44.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-25T21:03:44.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-25T21:03:44.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-25T21:03:44.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-25T21:03:44.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-25T21:03:44.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-portend-2.6-1.el8.noarch 2023-12-25T21:03:44.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-25T21:03:44.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-25T21:03:44.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-25T21:03:44.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-25T21:03:44.251 INFO:teuthology.orchestra.run.smithi061.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-25T21:03:44.252 INFO:teuthology.orchestra.run.smithi061.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-25T21:03:44.252 INFO:teuthology.orchestra.run.smithi061.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-25T21:03:44.252 INFO:teuthology.orchestra.run.smithi061.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-25T21:03:44.252 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-25T21:03:44.252 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-25T21:03:44.252 INFO:teuthology.orchestra.run.smithi061.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-25T21:03:44.252 INFO:teuthology.orchestra.run.smithi061.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-25T21:03:44.252 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:03:44.252 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:03:44.465 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-base 2023-12-25T21:03:45.179 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:03:45.180 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:03:45.180 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2023-12-25T21:03:45.180 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:03:45.180 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2023-12-25T21:03:45.180 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-12-25T21:03:45.180 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:45.180 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2023-12-25T21:03:45.180 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:03:45.180 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2023-12-25T21:03:45.180 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:45.181 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 71 k 2023-12-25T21:03:45.181 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 290 k 2023-12-25T21:03:45.181 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2023-12-25T21:03:45.223 INFO:teuthology.orchestra.run.smithi174.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.7 MB/s | 71 kB 00:00 2023-12-25T21:03:45.223 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:03:45.224 INFO:teuthology.orchestra.run.smithi174.stdout:Total 1.6 MB/s | 71 kB 00:00 2023-12-25T21:03:45.224 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2023-12-25T21:03:45.231 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2023-12-25T21:03:45.231 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2023-12-25T21:03:45.232 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2023-12-25T21:03:45.236 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2023-12-25T21:03:45.275 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2023-12-25T21:03:45.359 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:18 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:03:45.822 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-25T21:03:46.004 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-25T21:03:46.228 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-25T21:03:46.914 INFO:teuthology.orchestra.run.smithi061.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T21:03:46.942 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-25T21:03:46.943 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:46.943 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2023-12-25T21:03:46.943 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-12-25T21:03:46.943 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:46.943 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:03:47.150 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-immutable-object-cache 2023-12-25T21:03:47.160 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:03:47.161 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-12-25T21:03:47.161 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:03:47.259 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install cephadm 2023-12-25T21:03:48.013 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:21 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:03:48.152 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:21 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:03:49.715 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:03:49.716 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:03:49.716 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repo Size 2023-12-25T21:03:49.716 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:03:49.717 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2023-12-25T21:03:49.717 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-12-25T21:03:49.717 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:49.717 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2023-12-25T21:03:49.717 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:03:49.717 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2023-12-25T21:03:49.717 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:49.717 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 163 k 2023-12-25T21:03:49.717 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 587 k 2023-12-25T21:03:49.717 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2023-12-25T21:03:49.760 INFO:teuthology.orchestra.run.smithi174.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.8 MB/s | 163 kB 00:00 2023-12-25T21:03:49.761 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:03:49.761 INFO:teuthology.orchestra.run.smithi174.stdout:Total 3.7 MB/s | 163 kB 00:00 2023-12-25T21:03:49.761 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2023-12-25T21:03:49.771 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2023-12-25T21:03:49.771 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2023-12-25T21:03:49.818 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2023-12-25T21:03:49.821 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2023-12-25T21:03:49.931 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2023-12-25T21:03:49.968 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:03:49.968 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:03:49.968 INFO:teuthology.orchestra.run.smithi061.stdout: Package Architecture Version Repository Size 2023-12-25T21:03:49.968 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:03:49.968 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-12-25T21:03:49.968 INFO:teuthology.orchestra.run.smithi061.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-12-25T21:03:49.969 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:03:49.969 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-12-25T21:03:49.969 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:03:49.969 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-12-25T21:03:49.969 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:03:49.969 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 71 k 2023-12-25T21:03:49.969 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 290 k 2023-12-25T21:03:49.970 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-12-25T21:03:50.014 INFO:teuthology.orchestra.run.smithi061.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.6 MB/s | 71 kB 00:00 2023-12-25T21:03:50.015 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:03:50.015 INFO:teuthology.orchestra.run.smithi061.stdout:Total 1.5 MB/s | 71 kB 00:00 2023-12-25T21:03:50.015 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-12-25T21:03:50.023 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-12-25T21:03:50.023 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-12-25T21:03:50.025 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-12-25T21:03:50.028 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-12-25T21:03:50.075 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-12-25T21:03:50.159 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:03:50.569 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:03:50.588 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-25T21:03:50.798 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-25T21:03:51.035 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-25T21:03:51.775 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-25T21:03:51.775 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:03:51.775 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-12-25T21:03:51.775 INFO:teuthology.orchestra.run.smithi061.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-12-25T21:03:51.775 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:03:51.775 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:03:51.955 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-immutable-object-cache 2023-12-25T21:03:52.316 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:03:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:52.317 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2023-12-25T21:03:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:52.317 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:52.317 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:03:52.542 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr 2023-12-25T21:03:52.853 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:25 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:03:53.404 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:26 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:03:54.648 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:03:54.648 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:03:54.649 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repo Size 2023-12-25T21:03:54.649 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:03:54.649 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-12-25T21:03:54.649 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-12-25T21:03:54.649 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:03:54.649 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-12-25T21:03:54.649 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:03:54.649 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-12-25T21:03:54.649 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:03:54.650 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 163 k 2023-12-25T21:03:54.650 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 587 k 2023-12-25T21:03:54.650 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-12-25T21:03:54.694 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.6 MB/s | 163 kB 00:00 2023-12-25T21:03:54.694 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:03:54.695 INFO:teuthology.orchestra.run.smithi061.stdout:Total 3.5 MB/s | 163 kB 00:00 2023-12-25T21:03:54.695 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-12-25T21:03:54.707 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-12-25T21:03:54.707 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-12-25T21:03:54.757 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-12-25T21:03:54.760 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-12-25T21:03:54.866 INFO:teuthology.orchestra.run.smithi174.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T21:03:54.875 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-12-25T21:03:55.059 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:03:55.096 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:03:55.097 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2023-12-25T21:03:55.097 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:03:55.199 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-dashboard 2023-12-25T21:03:55.527 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:03:56.051 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:29 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:03:56.224 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:03:56.224 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:03:56.224 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-12-25T21:03:56.224 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-12-25T21:03:56.224 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:03:56.224 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:03:56.484 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-mgr 2023-12-25T21:03:57.376 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:30 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:03:57.745 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:03:57.746 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:03:57.746 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2023-12-25T21:03:57.746 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:03:57.746 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2023-12-25T21:03:57.746 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-12-25T21:03:57.747 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2023-12-25T21:03:57.747 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-12-25T21:03:57.747 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-12-25T21:03:57.747 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-12-25T21:03:57.747 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-25T21:03:57.747 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-25T21:03:57.747 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:57.747 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2023-12-25T21:03:57.747 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:03:57.747 INFO:teuthology.orchestra.run.smithi174.stdout:Install 6 Packages 2023-12-25T21:03:57.747 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:03:57.747 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 3.0 M 2023-12-25T21:03:57.747 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 15 M 2023-12-25T21:03:57.747 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2023-12-25T21:03:57.904 INFO:teuthology.orchestra.run.smithi174.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 391 kB/s | 24 kB 00:00 2023-12-25T21:03:57.929 INFO:teuthology.orchestra.run.smithi174.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 103 kB/s | 8.9 kB 00:00 2023-12-25T21:03:57.988 INFO:teuthology.orchestra.run.smithi174.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 18 MB/s | 2.7 MB 00:00 2023-12-25T21:03:58.098 INFO:teuthology.orchestra.run.smithi174.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 199 kB/s | 34 kB 00:00 2023-12-25T21:03:58.130 INFO:teuthology.orchestra.run.smithi174.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 189 kB/s | 43 kB 00:00 2023-12-25T21:03:58.180 INFO:teuthology.orchestra.run.smithi174.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.0 MB/s | 196 kB 00:00 2023-12-25T21:03:58.182 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:03:58.182 INFO:teuthology.orchestra.run.smithi174.stdout:Total 6.7 MB/s | 3.0 MB 00:00 2023-12-25T21:03:58.192 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2023-12-25T21:03:58.202 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2023-12-25T21:03:58.202 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2023-12-25T21:03:58.358 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2023-12-25T21:03:58.361 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2023-12-25T21:03:58.656 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2023-12-25T21:03:58.871 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-12-25T21:03:58.954 INFO:teuthology.orchestra.run.smithi061.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T21:03:59.141 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-12-25T21:03:59.209 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:03:59.210 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-12-25T21:03:59.210 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:03:59.320 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-mgr-dashboard 2023-12-25T21:03:59.351 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-12-25T21:03:59.489 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-12-25T21:04:00.211 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:33 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:00.470 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-12-25T21:04:00.730 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-25T21:04:01.002 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-25T21:04:01.003 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-12-25T21:04:01.003 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-12-25T21:04:01.003 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-12-25T21:04:01.003 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-12-25T21:04:01.003 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-12-25T21:04:01.707 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-12-25T21:04:01.707 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:01.707 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2023-12-25T21:04:01.707 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-12-25T21:04:01.707 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-12-25T21:04:01.707 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-12-25T21:04:01.707 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-25T21:04:01.708 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-25T21:04:01.708 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-25T21:04:01.708 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:01.708 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:04:01.846 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-25T21:04:02.009 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:04:02.010 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:04:02.010 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-12-25T21:04:02.010 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:04:02.010 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-12-25T21:04:02.010 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-12-25T21:04:02.010 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-12-25T21:04:02.010 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-12-25T21:04:02.010 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-12-25T21:04:02.011 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-12-25T21:04:02.011 INFO:teuthology.orchestra.run.smithi061.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-25T21:04:02.011 INFO:teuthology.orchestra.run.smithi061.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-25T21:04:02.011 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:02.011 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-12-25T21:04:02.011 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:04:02.011 INFO:teuthology.orchestra.run.smithi061.stdout:Install 6 Packages 2023-12-25T21:04:02.011 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:02.012 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 3.0 M 2023-12-25T21:04:02.012 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 15 M 2023-12-25T21:04:02.012 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-12-25T21:04:02.396 INFO:teuthology.orchestra.run.smithi061.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 434 kB/s | 24 kB 00:00 2023-12-25T21:04:02.421 INFO:teuthology.orchestra.run.smithi061.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 111 kB/s | 8.9 kB 00:00 2023-12-25T21:04:02.522 INFO:teuthology.orchestra.run.smithi061.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 15 MB/s | 2.7 MB 00:00 2023-12-25T21:04:02.639 INFO:teuthology.orchestra.run.smithi061.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 176 kB/s | 43 kB 00:00 2023-12-25T21:04:02.704 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:35 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:02.722 INFO:teuthology.orchestra.run.smithi061.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 112 kB/s | 34 kB 00:00 2023-12-25T21:04:02.898 INFO:teuthology.orchestra.run.smithi061.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 530 kB/s | 196 kB 00:00 2023-12-25T21:04:02.901 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:04:02.902 INFO:teuthology.orchestra.run.smithi061.stdout:Total 3.3 MB/s | 3.0 MB 00:00 2023-12-25T21:04:02.913 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-12-25T21:04:02.924 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-12-25T21:04:02.924 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-12-25T21:04:03.093 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-12-25T21:04:03.096 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-12-25T21:04:03.399 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-12-25T21:04:03.632 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-12-25T21:04:03.918 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-12-25T21:04:04.160 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-12-25T21:04:04.383 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:04:04.384 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================================ 2023-12-25T21:04:04.384 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2023-12-25T21:04:04.384 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================================ 2023-12-25T21:04:04.384 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2023-12-25T21:04:04.384 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-12-25T21:04:04.384 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2023-12-25T21:04:04.384 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-12-25T21:04:04.384 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-12-25T21:04:04.384 INFO:teuthology.orchestra.run.smithi174.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-12-25T21:04:04.385 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-12-25T21:04:04.385 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-12-25T21:04:04.385 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-12-25T21:04:04.385 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2023-12-25T21:04:04.385 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:04.385 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2023-12-25T21:04:04.385 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================================ 2023-12-25T21:04:04.385 INFO:teuthology.orchestra.run.smithi174.stdout:Install 8 Packages 2023-12-25T21:04:04.385 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:04.386 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 37 M 2023-12-25T21:04:04.386 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 210 M 2023-12-25T21:04:04.386 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2023-12-25T21:04:04.449 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-12-25T21:04:04.918 INFO:teuthology.orchestra.run.smithi174.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 59 MB/s | 8.6 MB 00:00 2023-12-25T21:04:05.476 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-12-25T21:04:05.618 INFO:teuthology.orchestra.run.smithi174.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.7 MB/s | 4.9 MB 00:00 2023-12-25T21:04:05.685 INFO:teuthology.orchestra.run.smithi174.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 5.2 MB/s | 4.7 MB 00:00 2023-12-25T21:04:05.710 INFO:teuthology.orchestra.run.smithi174.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2023-12-25T21:04:05.773 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-25T21:04:05.987 INFO:teuthology.orchestra.run.smithi174.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.5 MB/s | 3.7 MB 00:01 2023-12-25T21:04:06.053 INFO:teuthology.orchestra.run.smithi174.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.8 MB/s | 645 kB 00:00 2023-12-25T21:04:06.080 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-25T21:04:06.080 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-12-25T21:04:06.080 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-12-25T21:04:06.080 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-12-25T21:04:06.081 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-12-25T21:04:06.081 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-12-25T21:04:06.178 INFO:teuthology.orchestra.run.smithi174.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 899 kB/s | 172 kB 00:00 2023-12-25T21:04:06.553 INFO:teuthology.orchestra.run.smithi174.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 16 MB/s | 14 MB 00:00 2023-12-25T21:04:06.554 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:04:06.554 INFO:teuthology.orchestra.run.smithi174.stdout:Total 17 MB/s | 37 MB 00:02 2023-12-25T21:04:06.784 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2023-12-25T21:04:06.802 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2023-12-25T21:04:06.802 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2023-12-25T21:04:06.813 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-12-25T21:04:06.814 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:06.814 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-12-25T21:04:06.814 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-12-25T21:04:06.814 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-12-25T21:04:06.814 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-12-25T21:04:06.814 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-25T21:04:06.815 INFO:teuthology.orchestra.run.smithi061.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-25T21:04:06.815 INFO:teuthology.orchestra.run.smithi061.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-25T21:04:06.815 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:06.815 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:04:06.944 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-25T21:04:07.116 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2023-12-25T21:04:07.120 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2023-12-25T21:04:07.757 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2023-12-25T21:04:07.839 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:40 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:08.549 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-25T21:04:09.007 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-25T21:04:09.189 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-25T21:04:09.635 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:04:09.637 INFO:teuthology.orchestra.run.smithi061.stdout:============================================================================================ 2023-12-25T21:04:09.637 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-12-25T21:04:09.637 INFO:teuthology.orchestra.run.smithi061.stdout:============================================================================================ 2023-12-25T21:04:09.637 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-12-25T21:04:09.637 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-12-25T21:04:09.637 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-12-25T21:04:09.637 INFO:teuthology.orchestra.run.smithi061.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-12-25T21:04:09.637 INFO:teuthology.orchestra.run.smithi061.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-12-25T21:04:09.638 INFO:teuthology.orchestra.run.smithi061.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-12-25T21:04:09.638 INFO:teuthology.orchestra.run.smithi061.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-12-25T21:04:09.638 INFO:teuthology.orchestra.run.smithi061.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-12-25T21:04:09.638 INFO:teuthology.orchestra.run.smithi061.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-12-25T21:04:09.638 INFO:teuthology.orchestra.run.smithi061.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2023-12-25T21:04:09.638 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:09.638 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-12-25T21:04:09.638 INFO:teuthology.orchestra.run.smithi061.stdout:============================================================================================ 2023-12-25T21:04:09.638 INFO:teuthology.orchestra.run.smithi061.stdout:Install 8 Packages 2023-12-25T21:04:09.638 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:09.639 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 37 M 2023-12-25T21:04:09.639 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 210 M 2023-12-25T21:04:09.639 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-12-25T21:04:09.770 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-25T21:04:09.981 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-12-25T21:04:10.340 INFO:teuthology.orchestra.run.smithi061.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 18 MB/s | 8.6 MB 00:00 2023-12-25T21:04:10.594 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-12-25T21:04:10.780 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-25T21:04:11.048 INFO:teuthology.orchestra.run.smithi061.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.1 MB/s | 4.9 MB 00:01 2023-12-25T21:04:11.107 INFO:teuthology.orchestra.run.smithi061.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 3.8 MB/s | 4.7 MB 00:01 2023-12-25T21:04:11.140 INFO:teuthology.orchestra.run.smithi061.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2023-12-25T21:04:11.274 INFO:teuthology.orchestra.run.smithi061.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.0 MB/s | 3.7 MB 00:00 2023-12-25T21:04:11.407 INFO:teuthology.orchestra.run.smithi061.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.4 MB/s | 645 kB 00:00 2023-12-25T21:04:11.466 INFO:teuthology.orchestra.run.smithi061.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 900 kB/s | 172 kB 00:00 2023-12-25T21:04:11.500 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-25T21:04:11.867 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-25T21:04:11.916 INFO:teuthology.orchestra.run.smithi061.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 18 MB/s | 14 MB 00:00 2023-12-25T21:04:11.917 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:04:11.917 INFO:teuthology.orchestra.run.smithi061.stdout:Total 16 MB/s | 37 MB 00:02 2023-12-25T21:04:12.099 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-25T21:04:12.161 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-12-25T21:04:12.181 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-12-25T21:04:12.181 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-12-25T21:04:12.555 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-12-25T21:04:12.558 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-12-25T21:04:13.210 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-12-25T21:04:13.356 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-25T21:04:13.538 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-25T21:04:13.681 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-25T21:04:13.864 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-25T21:04:14.339 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-25T21:04:14.523 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-12-25T21:04:15.177 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-12-25T21:04:15.247 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2023-12-25T21:04:15.457 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-25T21:04:15.473 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-25T21:04:15.779 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-25T21:04:15.779 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-12-25T21:04:15.779 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-12-25T21:04:15.779 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-12-25T21:04:15.779 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-25T21:04:15.779 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-25T21:04:15.779 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2023-12-25T21:04:15.780 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-12-25T21:04:16.196 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-25T21:04:16.512 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-12-25T21:04:16.512 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:16.512 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2023-12-25T21:04:16.512 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-12-25T21:04:16.512 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-12-25T21:04:16.512 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-12-25T21:04:16.512 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-0.3.15-4.el8.x86_64 2023-12-25T21:04:16.513 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-12-25T21:04:16.513 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-25T21:04:16.513 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-25T21:04:16.513 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2023-12-25T21:04:16.513 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:16.513 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:04:16.547 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-25T21:04:16.627 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-rook 2023-12-25T21:04:16.790 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-25T21:04:17.488 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:50 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:18.679 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-25T21:04:19.155 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:04:19.156 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:04:19.156 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2023-12-25T21:04:19.156 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:04:19.157 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2023-12-25T21:04:19.157 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-12-25T21:04:19.157 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2023-12-25T21:04:19.157 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-25T21:04:19.157 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-25T21:04:19.157 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-25T21:04:19.157 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-12-25T21:04:19.157 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-12-25T21:04:19.157 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-25T21:04:19.157 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-12-25T21:04:19.157 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-12-25T21:04:19.157 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-12-25T21:04:19.158 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-12-25T21:04:19.158 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-25T21:04:19.158 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-25T21:04:19.158 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:19.158 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2023-12-25T21:04:19.158 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:04:19.158 INFO:teuthology.orchestra.run.smithi174.stdout:Install 13 Packages 2023-12-25T21:04:19.158 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:19.159 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 1.9 M 2023-12-25T21:04:19.159 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 24 M 2023-12-25T21:04:19.159 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2023-12-25T21:04:19.584 INFO:teuthology.orchestra.run.smithi174.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 519 kB/s | 27 kB 00:00 2023-12-25T21:04:19.767 INFO:teuthology.orchestra.run.smithi174.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 85 kB/s | 20 kB 00:00 2023-12-25T21:04:19.793 INFO:teuthology.orchestra.run.smithi174.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 102 kB/s | 27 kB 00:00 2023-12-25T21:04:19.943 INFO:teuthology.orchestra.run.smithi174.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 627 kB/s | 110 kB 00:00 2023-12-25T21:04:19.985 INFO:teuthology.orchestra.run.smithi174.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 315 kB/s | 126 kB 00:00 2023-12-25T21:04:20.010 INFO:teuthology.orchestra.run.smithi174.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 713 kB/s | 155 kB 00:00 2023-12-25T21:04:20.071 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2023-12-25T21:04:20.077 INFO:teuthology.orchestra.run.smithi174.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 359 kB/s | 33 kB 00:00 2023-12-25T21:04:20.094 INFO:teuthology.orchestra.run.smithi174.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 283 kB/s | 43 kB 00:00 2023-12-25T21:04:20.161 INFO:teuthology.orchestra.run.smithi174.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 975 kB/s | 81 kB 00:00 2023-12-25T21:04:20.178 INFO:teuthology.orchestra.run.smithi174.stdout:(10/13): python3-certifi-2018.10.15-7.el8.noarc 97 kB/s | 16 kB 00:00 2023-12-25T21:04:20.253 INFO:teuthology.orchestra.run.smithi174.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 697 kB/s | 64 kB 00:00 2023-12-25T21:04:20.320 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-25T21:04:20.328 INFO:teuthology.orchestra.run.smithi174.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 407 kB/s | 61 kB 00:00 2023-12-25T21:04:20.437 INFO:teuthology.orchestra.run.smithi174.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.4 MB/s | 1.2 MB 00:00 2023-12-25T21:04:20.437 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:04:20.437 INFO:teuthology.orchestra.run.smithi174.stdout:Total 1.5 MB/s | 1.9 MB 00:01 2023-12-25T21:04:20.497 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2023-12-25T21:04:20.513 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2023-12-25T21:04:20.513 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2023-12-25T21:04:20.600 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2023-12-25T21:04:20.603 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2023-12-25T21:04:20.662 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-25T21:04:20.662 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-12-25T21:04:20.662 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-12-25T21:04:20.663 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-12-25T21:04:20.663 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-25T21:04:20.663 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-25T21:04:20.663 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2023-12-25T21:04:20.663 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-12-25T21:04:21.114 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2023-12-25T21:04:21.380 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-25T21:04:21.497 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-12-25T21:04:21.498 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:21.498 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-12-25T21:04:21.498 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-12-25T21:04:21.498 INFO:teuthology.orchestra.run.smithi061.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-12-25T21:04:21.498 INFO:teuthology.orchestra.run.smithi061.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-12-25T21:04:21.498 INFO:teuthology.orchestra.run.smithi061.stdout: openblas-0.3.15-4.el8.x86_64 2023-12-25T21:04:21.498 INFO:teuthology.orchestra.run.smithi061.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-12-25T21:04:21.498 INFO:teuthology.orchestra.run.smithi061.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-25T21:04:21.498 INFO:teuthology.orchestra.run.smithi061.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-25T21:04:21.498 INFO:teuthology.orchestra.run.smithi061.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2023-12-25T21:04:21.498 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:21.498 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:04:21.584 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-12-25T21:04:21.622 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-mgr-rook 2023-12-25T21:04:21.825 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-12-25T21:04:22.036 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-12-25T21:04:22.298 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-12-25T21:04:22.516 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:55 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:22.521 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-12-25T21:04:22.825 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-12-25T21:04:23.085 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-25T21:04:24.201 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-25T21:04:24.317 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:04:24.319 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:04:24.319 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-12-25T21:04:24.319 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:04:24.319 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-12-25T21:04:24.320 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-12-25T21:04:24.320 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-12-25T21:04:24.320 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-25T21:04:24.320 INFO:teuthology.orchestra.run.smithi061.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-25T21:04:24.320 INFO:teuthology.orchestra.run.smithi061.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-25T21:04:24.320 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-12-25T21:04:24.320 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-12-25T21:04:24.320 INFO:teuthology.orchestra.run.smithi061.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-25T21:04:24.320 INFO:teuthology.orchestra.run.smithi061.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-12-25T21:04:24.320 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-12-25T21:04:24.320 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-12-25T21:04:24.320 INFO:teuthology.orchestra.run.smithi061.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-12-25T21:04:24.320 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-25T21:04:24.320 INFO:teuthology.orchestra.run.smithi061.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-25T21:04:24.321 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:24.321 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-12-25T21:04:24.321 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:04:24.321 INFO:teuthology.orchestra.run.smithi061.stdout:Install 13 Packages 2023-12-25T21:04:24.321 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:24.321 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 1.9 M 2023-12-25T21:04:24.321 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 24 M 2023-12-25T21:04:24.321 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-12-25T21:04:24.486 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-12-25T21:04:24.688 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-25T21:04:24.764 INFO:teuthology.orchestra.run.smithi061.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 632 kB/s | 27 kB 00:00 2023-12-25T21:04:24.895 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-25T21:04:24.981 INFO:teuthology.orchestra.run.smithi061.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 102 kB/s | 27 kB 00:00 2023-12-25T21:04:24.998 INFO:teuthology.orchestra.run.smithi061.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 72 kB/s | 20 kB 00:00 2023-12-25T21:04:25.061 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-25T21:04:25.166 INFO:teuthology.orchestra.run.smithi061.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 315 kB/s | 126 kB 00:00 2023-12-25T21:04:25.190 INFO:teuthology.orchestra.run.smithi061.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 526 kB/s | 110 kB 00:00 2023-12-25T21:04:25.232 INFO:teuthology.orchestra.run.smithi061.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 662 kB/s | 155 kB 00:00 2023-12-25T21:04:25.291 INFO:teuthology.orchestra.run.smithi061.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 328 kB/s | 33 kB 00:00 2023-12-25T21:04:25.316 INFO:teuthology.orchestra.run.smithi061.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 285 kB/s | 43 kB 00:00 2023-12-25T21:04:25.342 INFO:teuthology.orchestra.run.smithi061.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 150 kB/s | 16 kB 00:00 2023-12-25T21:04:25.366 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-25T21:04:25.366 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-12-25T21:04:25.366 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-12-25T21:04:25.366 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-12-25T21:04:25.366 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-12-25T21:04:25.367 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-12-25T21:04:25.367 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-25T21:04:25.367 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-25T21:04:25.367 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-12-25T21:04:25.367 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-25T21:04:25.367 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-12-25T21:04:25.367 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-12-25T21:04:25.367 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-12-25T21:04:25.375 INFO:teuthology.orchestra.run.smithi061.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 970 kB/s | 81 kB 00:00 2023-12-25T21:04:25.434 INFO:teuthology.orchestra.run.smithi061.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2023-12-25T21:04:25.493 INFO:teuthology.orchestra.run.smithi061.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 427 kB/s | 64 kB 00:00 2023-12-25T21:04:25.526 INFO:teuthology.orchestra.run.smithi061.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.6 MB/s | 1.2 MB 00:00 2023-12-25T21:04:25.526 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:04:25.527 INFO:teuthology.orchestra.run.smithi061.stdout:Total 1.6 MB/s | 1.9 MB 00:01 2023-12-25T21:04:25.589 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-12-25T21:04:25.605 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-12-25T21:04:25.605 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-12-25T21:04:25.701 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-12-25T21:04:25.705 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-12-25T21:04:26.141 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-12-25T21:04:26.141 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:26.141 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2023-12-25T21:04:26.141 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-12-25T21:04:26.141 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-25T21:04:26.142 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-25T21:04:26.142 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-25T21:04:26.142 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-25T21:04:26.142 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-25T21:04:26.142 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-25T21:04:26.142 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-25T21:04:26.142 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-25T21:04:26.142 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-25T21:04:26.142 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-25T21:04:26.142 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-25T21:04:26.142 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-25T21:04:26.142 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:26.142 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:04:26.263 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-12-25T21:04:26.267 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-cephadm 2023-12-25T21:04:26.584 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-25T21:04:26.800 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-12-25T21:04:27.058 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-12-25T21:04:27.127 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:00 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:27.278 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-12-25T21:04:27.530 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-12-25T21:04:27.815 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-12-25T21:04:28.135 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-12-25T21:04:28.396 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-25T21:04:28.817 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:04:28.818 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:04:28.818 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2023-12-25T21:04:28.818 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:04:28.818 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2023-12-25T21:04:28.818 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-12-25T21:04:28.818 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2023-12-25T21:04:28.818 INFO:teuthology.orchestra.run.smithi174.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-25T21:04:28.818 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-12-25T21:04:28.818 INFO:teuthology.orchestra.run.smithi174.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-25T21:04:28.818 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-12-25T21:04:28.818 INFO:teuthology.orchestra.run.smithi174.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-25T21:04:28.818 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:28.818 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2023-12-25T21:04:28.819 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:04:28.819 INFO:teuthology.orchestra.run.smithi174.stdout:Install 6 Packages 2023-12-25T21:04:28.819 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:28.819 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 5.8 M 2023-12-25T21:04:28.819 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 24 M 2023-12-25T21:04:28.819 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2023-12-25T21:04:28.976 INFO:teuthology.orchestra.run.smithi174.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.5 MB/s | 89 kB 00:00 2023-12-25T21:04:29.210 INFO:teuthology.orchestra.run.smithi174.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 91 kB/s | 21 kB 00:00 2023-12-25T21:04:29.524 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-25T21:04:29.634 INFO:teuthology.orchestra.run.smithi174.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 753 kB/s | 538 kB 00:00 2023-12-25T21:04:29.736 INFO:teuthology.orchestra.run.smithi174.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 676 kB/s | 355 kB 00:00 2023-12-25T21:04:29.752 INFO:teuthology.orchestra.run.smithi174.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 415 kB/s | 49 kB 00:00 2023-12-25T21:04:29.819 INFO:teuthology.orchestra.run.smithi174.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 5.3 MB/s | 4.8 MB 00:00 2023-12-25T21:04:29.820 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:04:29.820 INFO:teuthology.orchestra.run.smithi174.stdout:Total 5.8 MB/s | 5.8 MB 00:01 2023-12-25T21:04:29.836 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-12-25T21:04:29.874 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2023-12-25T21:04:29.884 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2023-12-25T21:04:29.884 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2023-12-25T21:04:30.000 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2023-12-25T21:04:30.003 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2023-12-25T21:04:30.063 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-25T21:04:30.304 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-25T21:04:30.306 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2023-12-25T21:04:30.504 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-25T21:04:30.551 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-12-25T21:04:30.805 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-25T21:04:30.806 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-12-25T21:04:30.806 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-12-25T21:04:30.806 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-12-25T21:04:30.806 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-12-25T21:04:30.806 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-12-25T21:04:30.806 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-25T21:04:30.806 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-25T21:04:30.806 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-12-25T21:04:30.806 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-25T21:04:30.806 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-12-25T21:04:30.806 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-12-25T21:04:30.806 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-12-25T21:04:30.813 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-12-25T21:04:31.595 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-12-25T21:04:31.595 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:31.595 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-12-25T21:04:31.595 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-12-25T21:04:31.595 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-25T21:04:31.595 INFO:teuthology.orchestra.run.smithi061.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-25T21:04:31.596 INFO:teuthology.orchestra.run.smithi061.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-25T21:04:31.596 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-25T21:04:31.596 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-25T21:04:31.596 INFO:teuthology.orchestra.run.smithi061.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-25T21:04:31.596 INFO:teuthology.orchestra.run.smithi061.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-25T21:04:31.596 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-25T21:04:31.596 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-25T21:04:31.596 INFO:teuthology.orchestra.run.smithi061.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-25T21:04:31.596 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-25T21:04:31.596 INFO:teuthology.orchestra.run.smithi061.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-25T21:04:31.596 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:31.596 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:04:31.737 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-12-25T21:04:31.756 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-mgr-cephadm 2023-12-25T21:04:32.100 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-12-25T21:04:32.350 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-12-25T21:04:32.560 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-25T21:04:32.656 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:05 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:32.859 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-25T21:04:32.860 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-12-25T21:04:32.860 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-12-25T21:04:32.860 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-12-25T21:04:32.860 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-12-25T21:04:32.860 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-12-25T21:04:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-12-25T21:04:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:33.576 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2023-12-25T21:04:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-12-25T21:04:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-12-25T21:04:33.576 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-12-25T21:04:33.577 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:33.577 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:04:33.727 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-fuse 2023-12-25T21:04:34.474 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:04:34.476 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:04:34.476 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-12-25T21:04:34.476 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:04:34.476 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-12-25T21:04:34.476 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-12-25T21:04:34.476 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-12-25T21:04:34.476 INFO:teuthology.orchestra.run.smithi061.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-25T21:04:34.476 INFO:teuthology.orchestra.run.smithi061.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-12-25T21:04:34.476 INFO:teuthology.orchestra.run.smithi061.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-25T21:04:34.476 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-12-25T21:04:34.476 INFO:teuthology.orchestra.run.smithi061.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-25T21:04:34.477 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:34.477 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-12-25T21:04:34.477 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:04:34.477 INFO:teuthology.orchestra.run.smithi061.stdout:Install 6 Packages 2023-12-25T21:04:34.477 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:34.477 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 5.8 M 2023-12-25T21:04:34.477 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 24 M 2023-12-25T21:04:34.478 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-12-25T21:04:34.581 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:07 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:35.083 INFO:teuthology.orchestra.run.smithi061.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 89 kB 00:00 2023-12-25T21:04:35.216 INFO:teuthology.orchestra.run.smithi061.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 159 kB/s | 21 kB 00:00 2023-12-25T21:04:35.417 INFO:teuthology.orchestra.run.smithi061.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.7 MB/s | 355 kB 00:00 2023-12-25T21:04:35.476 INFO:teuthology.orchestra.run.smithi061.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 838 kB/s | 49 kB 00:00 2023-12-25T21:04:35.626 INFO:teuthology.orchestra.run.smithi061.stdout:(5/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 909 kB/s | 538 kB 00:00 2023-12-25T21:04:35.948 INFO:teuthology.orchestra.run.smithi061.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 5.2 MB/s | 4.8 MB 00:00 2023-12-25T21:04:35.949 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:04:35.949 INFO:teuthology.orchestra.run.smithi061.stdout:Total 3.9 MB/s | 5.8 MB 00:01 2023-12-25T21:04:36.005 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-12-25T21:04:36.016 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-12-25T21:04:36.016 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-12-25T21:04:36.148 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-12-25T21:04:36.151 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-12-25T21:04:36.275 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:04:36.276 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:04:36.276 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2023-12-25T21:04:36.276 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:04:36.276 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2023-12-25T21:04:36.276 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-12-25T21:04:36.276 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2023-12-25T21:04:36.276 INFO:teuthology.orchestra.run.smithi174.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-12-25T21:04:36.276 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:36.277 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2023-12-25T21:04:36.277 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:04:36.277 INFO:teuthology.orchestra.run.smithi174.stdout:Install 2 Packages 2023-12-25T21:04:36.277 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:36.277 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 968 k 2023-12-25T21:04:36.277 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 3.2 M 2023-12-25T21:04:36.277 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2023-12-25T21:04:36.472 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-12-25T21:04:36.513 INFO:teuthology.orchestra.run.smithi174.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 17 MB/s | 885 kB 00:00 2023-12-25T21:04:36.639 INFO:teuthology.orchestra.run.smithi174.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 474 kB/s | 83 kB 00:00 2023-12-25T21:04:36.639 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:04:36.639 INFO:teuthology.orchestra.run.smithi174.stdout:Total 2.6 MB/s | 968 kB 00:00 2023-12-25T21:04:36.649 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2023-12-25T21:04:36.670 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2023-12-25T21:04:36.670 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2023-12-25T21:04:36.756 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2023-12-25T21:04:36.758 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-12-25T21:04:36.759 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2023-12-25T21:04:36.941 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2023-12-25T21:04:37.067 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-12-25T21:04:37.235 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-25T21:04:37.433 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-25T21:04:37.849 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-25T21:04:37.849 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-12-25T21:04:38.068 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-12-25T21:04:38.416 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-12-25T21:04:38.573 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-25T21:04:38.573 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:38.573 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2023-12-25T21:04:38.573 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-25T21:04:38.573 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:38.573 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:04:38.644 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-12-25T21:04:38.787 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install librados-devel 2023-12-25T21:04:38.869 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-25T21:04:39.201 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-25T21:04:39.202 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-12-25T21:04:39.202 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-12-25T21:04:39.202 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-12-25T21:04:39.202 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-12-25T21:04:39.202 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-12-25T21:04:39.650 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:12 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:40.024 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-12-25T21:04:40.025 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:40.025 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-12-25T21:04:40.025 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-12-25T21:04:40.025 INFO:teuthology.orchestra.run.smithi061.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-12-25T21:04:40.025 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-12-25T21:04:40.025 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:40.025 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:04:40.153 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-fuse 2023-12-25T21:04:41.041 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:14 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:41.346 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:04:41.347 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:04:41.347 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2023-12-25T21:04:41.347 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:04:41.347 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2023-12-25T21:04:41.347 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-12-25T21:04:41.347 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:41.347 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2023-12-25T21:04:41.347 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:04:41.347 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2023-12-25T21:04:41.347 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:41.348 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 131 k 2023-12-25T21:04:41.348 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 439 k 2023-12-25T21:04:41.348 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2023-12-25T21:04:41.391 INFO:teuthology.orchestra.run.smithi174.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 3.0 MB/s | 131 kB 00:00 2023-12-25T21:04:41.391 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:04:41.392 INFO:teuthology.orchestra.run.smithi174.stdout:Total 2.9 MB/s | 131 kB 00:00 2023-12-25T21:04:41.392 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2023-12-25T21:04:41.397 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2023-12-25T21:04:41.397 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2023-12-25T21:04:41.448 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2023-12-25T21:04:41.451 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2023-12-25T21:04:41.588 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2023-12-25T21:04:41.833 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:04:42.111 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:04:42.834 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:04:42.835 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:04:42.835 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:42.835 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2023-12-25T21:04:42.835 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-12-25T21:04:42.835 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:42.835 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:04:42.836 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:04:42.836 INFO:teuthology.orchestra.run.smithi061.stdout: Package Architecture Version Repository Size 2023-12-25T21:04:42.836 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:04:42.836 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-12-25T21:04:42.836 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-12-25T21:04:42.836 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-12-25T21:04:42.836 INFO:teuthology.orchestra.run.smithi061.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-12-25T21:04:42.836 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:42.836 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-12-25T21:04:42.837 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:04:42.837 INFO:teuthology.orchestra.run.smithi061.stdout:Install 2 Packages 2023-12-25T21:04:42.837 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:42.837 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 968 k 2023-12-25T21:04:42.837 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 3.2 M 2023-12-25T21:04:42.837 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-12-25T21:04:42.928 INFO:teuthology.orchestra.run.smithi061.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 17 MB/s | 885 kB 00:00 2023-12-25T21:04:43.048 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install libcephfs2 2023-12-25T21:04:43.054 INFO:teuthology.orchestra.run.smithi061.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 470 kB/s | 83 kB 00:00 2023-12-25T21:04:43.054 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:04:43.055 INFO:teuthology.orchestra.run.smithi061.stdout:Total 4.3 MB/s | 968 kB 00:00 2023-12-25T21:04:43.065 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-12-25T21:04:43.087 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-12-25T21:04:43.087 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-12-25T21:04:43.179 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-12-25T21:04:43.182 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-12-25T21:04:43.366 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-12-25T21:04:43.691 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-25T21:04:43.907 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:16 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:43.936 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-25T21:04:44.358 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-25T21:04:44.358 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-12-25T21:04:45.084 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-25T21:04:45.084 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:45.084 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-12-25T21:04:45.084 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-25T21:04:45.084 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:45.084 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:04:45.356 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install librados-devel 2023-12-25T21:04:45.361 INFO:teuthology.orchestra.run.smithi174.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T21:04:45.590 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:04:45.592 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2023-12-25T21:04:45.592 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:04:45.690 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install libcephfs-devel 2023-12-25T21:04:46.257 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:19 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:46.554 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:19 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:48.102 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:04:48.102 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:04:48.102 INFO:teuthology.orchestra.run.smithi061.stdout: Package Architecture Version Repository Size 2023-12-25T21:04:48.102 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:04:48.102 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-12-25T21:04:48.103 INFO:teuthology.orchestra.run.smithi061.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-12-25T21:04:48.103 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:48.103 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-12-25T21:04:48.103 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:04:48.103 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-12-25T21:04:48.103 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:48.103 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 131 k 2023-12-25T21:04:48.103 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 439 k 2023-12-25T21:04:48.104 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-12-25T21:04:48.151 INFO:teuthology.orchestra.run.smithi061.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.7 MB/s | 131 kB 00:00 2023-12-25T21:04:48.151 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:04:48.151 INFO:teuthology.orchestra.run.smithi061.stdout:Total 2.7 MB/s | 131 kB 00:00 2023-12-25T21:04:48.151 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-12-25T21:04:48.157 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-12-25T21:04:48.157 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-12-25T21:04:48.210 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-12-25T21:04:48.213 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-12-25T21:04:48.248 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:04:48.249 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:04:48.249 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2023-12-25T21:04:48.249 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:04:48.249 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2023-12-25T21:04:48.249 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-12-25T21:04:48.249 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:48.249 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2023-12-25T21:04:48.249 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:04:48.250 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2023-12-25T21:04:48.250 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:48.250 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 24 k 2023-12-25T21:04:48.250 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 94 k 2023-12-25T21:04:48.250 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2023-12-25T21:04:48.296 INFO:teuthology.orchestra.run.smithi174.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 532 kB/s | 24 kB 00:00 2023-12-25T21:04:48.296 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:04:48.297 INFO:teuthology.orchestra.run.smithi174.stdout:Total 518 kB/s | 24 kB 00:00 2023-12-25T21:04:48.297 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2023-12-25T21:04:48.300 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2023-12-25T21:04:48.300 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2023-12-25T21:04:48.303 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2023-12-25T21:04:48.305 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2023-12-25T21:04:48.353 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2023-12-25T21:04:48.357 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-12-25T21:04:48.573 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:04:48.643 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:04:48.843 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:04:48.950 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:04:49.505 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:04:49.505 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:49.505 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2023-12-25T21:04:49.505 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-12-25T21:04:49.506 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:04:49.506 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:04:49.628 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install librados2 2023-12-25T21:04:49.665 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:04:49.666 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:49.666 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-12-25T21:04:49.666 INFO:teuthology.orchestra.run.smithi061.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-12-25T21:04:49.666 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:49.666 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:04:49.910 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install libcephfs2 2023-12-25T21:04:50.482 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:23 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:50.802 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:23 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:51.949 INFO:teuthology.orchestra.run.smithi174.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T21:04:52.174 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:04:52.175 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2023-12-25T21:04:52.175 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:04:52.282 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install librbd1 2023-12-25T21:04:52.357 INFO:teuthology.orchestra.run.smithi061.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T21:04:52.601 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:04:52.602 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-12-25T21:04:52.602 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:04:52.715 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install libcephfs-devel 2023-12-25T21:04:53.135 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:26 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:53.612 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:26 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:54.585 INFO:teuthology.orchestra.run.smithi174.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T21:04:54.811 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:04:54.812 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2023-12-25T21:04:54.812 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:04:54.932 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-rados 2023-12-25T21:04:55.410 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:04:55.411 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:04:55.411 INFO:teuthology.orchestra.run.smithi061.stdout: Package Architecture Version Repository Size 2023-12-25T21:04:55.411 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:04:55.411 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-12-25T21:04:55.411 INFO:teuthology.orchestra.run.smithi061.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-12-25T21:04:55.411 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:55.412 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-12-25T21:04:55.412 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:04:55.412 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-12-25T21:04:55.412 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:55.412 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 24 k 2023-12-25T21:04:55.412 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 94 k 2023-12-25T21:04:55.412 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-12-25T21:04:55.463 INFO:teuthology.orchestra.run.smithi061.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 483 kB/s | 24 kB 00:00 2023-12-25T21:04:55.463 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:04:55.464 INFO:teuthology.orchestra.run.smithi061.stdout:Total 471 kB/s | 24 kB 00:00 2023-12-25T21:04:55.464 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-12-25T21:04:55.467 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-12-25T21:04:55.467 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-12-25T21:04:55.470 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-12-25T21:04:55.473 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-12-25T21:04:55.521 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-12-25T21:04:55.767 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:04:55.791 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:28 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:56.047 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:04:56.740 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:04:56.740 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:56.740 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-12-25T21:04:56.740 INFO:teuthology.orchestra.run.smithi061.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-12-25T21:04:56.740 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:04:56.740 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:04:56.862 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install librados2 2023-12-25T21:04:57.247 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T21:04:57.474 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:04:57.475 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2023-12-25T21:04:57.476 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:04:57.579 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-rgw 2023-12-25T21:04:57.759 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:30 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:58.437 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:31 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:04:59.312 INFO:teuthology.orchestra.run.smithi061.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T21:04:59.557 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:04:59.558 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-12-25T21:04:59.558 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:04:59.687 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install librbd1 2023-12-25T21:04:59.878 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T21:05:00.106 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:05:00.107 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2023-12-25T21:05:00.107 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:05:00.222 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-cephfs 2023-12-25T21:05:00.581 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:33 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:05:01.085 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:34 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:05:02.153 INFO:teuthology.orchestra.run.smithi061.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T21:05:02.397 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:05:02.398 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-12-25T21:05:02.398 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:05:02.502 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install python3-rados 2023-12-25T21:05:02.550 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T21:05:02.777 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:05:02.778 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2023-12-25T21:05:02.778 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:05:03.142 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-rbd 2023-12-25T21:05:03.393 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:36 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:05:03.995 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:36 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:05:04.944 INFO:teuthology.orchestra.run.smithi061.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T21:05:05.188 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:05:05.189 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-12-25T21:05:05.189 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:05:05.300 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install python3-rgw 2023-12-25T21:05:05.462 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T21:05:05.692 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:05:05.693 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2023-12-25T21:05:05.693 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:05:05.803 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install rbd-fuse 2023-12-25T21:05:06.206 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:39 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:05:06.666 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:39 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:05:07.778 INFO:teuthology.orchestra.run.smithi061.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T21:05:08.019 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:05:08.021 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-12-25T21:05:08.021 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:05:08.128 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install python3-cephfs 2023-12-25T21:05:08.361 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:05:08.362 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:05:08.362 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2023-12-25T21:05:08.362 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:05:08.362 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2023-12-25T21:05:08.362 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-12-25T21:05:08.362 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:05:08.362 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2023-12-25T21:05:08.362 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:05:08.362 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2023-12-25T21:05:08.363 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:05:08.363 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 89 k 2023-12-25T21:05:08.363 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 298 k 2023-12-25T21:05:08.363 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2023-12-25T21:05:08.419 INFO:teuthology.orchestra.run.smithi174.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.6 MB/s | 89 kB 00:00 2023-12-25T21:05:08.419 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:05:08.420 INFO:teuthology.orchestra.run.smithi174.stdout:Total 1.5 MB/s | 89 kB 00:00 2023-12-25T21:05:08.420 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2023-12-25T21:05:08.430 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2023-12-25T21:05:08.430 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2023-12-25T21:05:08.479 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2023-12-25T21:05:08.482 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2023-12-25T21:05:08.586 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2023-12-25T21:05:08.812 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:05:09.029 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:42 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:05:09.108 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:05:09.815 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:05:09.815 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:05:09.815 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2023-12-25T21:05:09.816 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-12-25T21:05:09.816 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:05:09.816 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:05:10.030 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install rbd-mirror 2023-12-25T21:05:10.629 INFO:teuthology.orchestra.run.smithi061.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T21:05:10.876 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:05:10.877 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-12-25T21:05:10.877 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:05:10.897 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:43 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:05:10.989 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install python3-rbd 2023-12-25T21:05:11.887 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:44 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:05:12.582 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:05:12.583 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:05:12.583 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2023-12-25T21:05:12.583 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:05:12.583 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2023-12-25T21:05:12.583 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-25T21:05:12.583 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:05:12.583 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2023-12-25T21:05:12.583 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:05:12.583 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2023-12-25T21:05:12.583 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:05:12.584 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 3.7 M 2023-12-25T21:05:12.584 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 18 M 2023-12-25T21:05:12.584 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2023-12-25T21:05:12.694 INFO:teuthology.orchestra.run.smithi174.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 34 MB/s | 3.7 MB 00:00 2023-12-25T21:05:12.698 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:05:12.698 INFO:teuthology.orchestra.run.smithi174.stdout:Total 33 MB/s | 3.7 MB 00:00 2023-12-25T21:05:12.698 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2023-12-25T21:05:12.707 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2023-12-25T21:05:12.707 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2023-12-25T21:05:12.771 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2023-12-25T21:05:12.774 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2023-12-25T21:05:13.236 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2023-12-25T21:05:13.463 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:05:13.466 INFO:teuthology.orchestra.run.smithi061.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-25T21:05:13.709 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:05:13.710 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-12-25T21:05:13.710 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:05:13.822 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install rbd-fuse 2023-12-25T21:05:13.899 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:05:14.583 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:05:14.584 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:05:14.584 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2023-12-25T21:05:14.584 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-12-25T21:05:14.584 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:05:14.584 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:05:14.719 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:47 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:05:14.770 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install rbd-nbd 2023-12-25T21:05:15.637 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:48 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:05:16.543 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:05:16.543 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:05:16.543 INFO:teuthology.orchestra.run.smithi061.stdout: Package Architecture Version Repository Size 2023-12-25T21:05:16.543 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:05:16.544 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-12-25T21:05:16.544 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-12-25T21:05:16.544 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:05:16.544 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-12-25T21:05:16.544 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:05:16.544 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-12-25T21:05:16.544 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:05:16.545 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 89 k 2023-12-25T21:05:16.545 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 298 k 2023-12-25T21:05:16.545 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-12-25T21:05:16.603 INFO:teuthology.orchestra.run.smithi061.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 89 kB 00:00 2023-12-25T21:05:16.603 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:05:16.603 INFO:teuthology.orchestra.run.smithi061.stdout:Total 1.5 MB/s | 89 kB 00:00 2023-12-25T21:05:16.604 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-12-25T21:05:16.614 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-12-25T21:05:16.614 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-12-25T21:05:16.668 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-12-25T21:05:16.671 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-12-25T21:05:16.782 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-12-25T21:05:17.023 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:05:17.331 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:05:17.331 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-12-25T21:05:17.332 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:05:17.332 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2023-12-25T21:05:17.333 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:05:17.333 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2023-12-25T21:05:17.333 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-12-25T21:05:17.333 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:05:17.333 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2023-12-25T21:05:17.333 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2023-12-25T21:05:17.333 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2023-12-25T21:05:17.333 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:05:17.334 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 173 k 2023-12-25T21:05:17.334 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 649 k 2023-12-25T21:05:17.334 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2023-12-25T21:05:17.381 INFO:teuthology.orchestra.run.smithi174.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.6 MB/s | 173 kB 00:00 2023-12-25T21:05:17.381 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:05:17.381 INFO:teuthology.orchestra.run.smithi174.stdout:Total 3.5 MB/s | 173 kB 00:00 2023-12-25T21:05:17.382 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2023-12-25T21:05:17.392 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2023-12-25T21:05:17.392 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2023-12-25T21:05:17.441 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2023-12-25T21:05:17.444 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2023-12-25T21:05:17.559 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2023-12-25T21:05:17.764 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:05:18.034 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:05:18.066 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:05:18.067 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:05:18.067 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-12-25T21:05:18.067 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-12-25T21:05:18.067 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:05:18.067 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:05:18.313 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install rbd-mirror 2023-12-25T21:05:18.708 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:05:18.708 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:05:18.709 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2023-12-25T21:05:18.709 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-12-25T21:05:18.709 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:05:18.709 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-12-25T21:05:18.986 DEBUG:teuthology.parallel:result is None 2023-12-25T21:05:19.220 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:52 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:05:21.022 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:05:21.023 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:05:21.023 INFO:teuthology.orchestra.run.smithi061.stdout: Package Architecture Version Repository Size 2023-12-25T21:05:21.023 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:05:21.023 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-12-25T21:05:21.023 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-25T21:05:21.023 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:05:21.023 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-12-25T21:05:21.023 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:05:21.024 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-12-25T21:05:21.024 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:05:21.024 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 3.7 M 2023-12-25T21:05:21.024 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 18 M 2023-12-25T21:05:21.024 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-12-25T21:05:21.105 INFO:teuthology.orchestra.run.smithi061.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 46 MB/s | 3.7 MB 00:00 2023-12-25T21:05:21.105 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:05:21.106 INFO:teuthology.orchestra.run.smithi061.stdout:Total 45 MB/s | 3.7 MB 00:00 2023-12-25T21:05:21.106 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-12-25T21:05:21.119 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-12-25T21:05:21.120 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-12-25T21:05:21.187 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-12-25T21:05:21.190 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-12-25T21:05:21.711 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-12-25T21:05:21.964 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:05:22.405 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:05:23.129 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:05:23.129 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:05:23.130 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-12-25T21:05:23.130 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-12-25T21:05:23.130 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:05:23.130 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:05:23.360 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install rbd-nbd 2023-12-25T21:05:24.263 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:57 ago on Mon 25 Dec 2023 09:02:27 PM UTC. 2023-12-25T21:05:26.065 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-12-25T21:05:26.066 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:05:26.066 INFO:teuthology.orchestra.run.smithi061.stdout: Package Architecture Version Repository Size 2023-12-25T21:05:26.066 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:05:26.066 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-12-25T21:05:26.066 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-12-25T21:05:26.066 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:05:26.066 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-12-25T21:05:26.066 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-12-25T21:05:26.066 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-12-25T21:05:26.066 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:05:26.067 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 173 k 2023-12-25T21:05:26.067 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 649 k 2023-12-25T21:05:26.067 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-12-25T21:05:26.138 INFO:teuthology.orchestra.run.smithi061.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 173 kB 00:00 2023-12-25T21:05:26.138 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-12-25T21:05:26.138 INFO:teuthology.orchestra.run.smithi061.stdout:Total 2.4 MB/s | 173 kB 00:00 2023-12-25T21:05:26.138 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-12-25T21:05:26.150 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-12-25T21:05:26.150 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-12-25T21:05:26.204 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-12-25T21:05:26.207 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-12-25T21:05:26.445 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-12-25T21:05:26.667 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:05:26.990 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:05:27.692 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-25T21:05:27.692 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:05:27.692 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-12-25T21:05:27.692 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-12-25T21:05:27.692 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-12-25T21:05:27.692 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-12-25T21:05:27.872 DEBUG:teuthology.parallel:result is None 2023-12-25T21:05:27.873 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-25T21:05:27.873 INFO:teuthology.packaging:ref: None 2023-12-25T21:05:27.873 INFO:teuthology.packaging:tag: v16.2.4 2023-12-25T21:05:27.873 INFO:teuthology.packaging:branch: None 2023-12-25T21:05:27.873 INFO:teuthology.packaging:sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T21:05:27.873 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-25T21:05:28.093 DEBUG:teuthology.orchestra.run.smithi061:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-25T21:05:28.117 INFO:teuthology.orchestra.run.smithi061.stdout:16.2.4-0.el8 2023-12-25T21:05:28.117 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-12-25T21:05:28.118 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-12-25T21:05:28.119 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-25T21:05:28.119 INFO:teuthology.packaging:ref: None 2023-12-25T21:05:28.119 INFO:teuthology.packaging:tag: v16.2.4 2023-12-25T21:05:28.119 INFO:teuthology.packaging:branch: None 2023-12-25T21:05:28.119 INFO:teuthology.packaging:sha1: b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2 2023-12-25T21:05:28.120 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-25T21:05:28.340 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-25T21:05:28.367 INFO:teuthology.orchestra.run.smithi174.stdout:16.2.4-0.el8 2023-12-25T21:05:28.367 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-12-25T21:05:28.367 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-12-25T21:05:28.369 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-25T21:05:28.369 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-12-25T21:05:28.369 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-25T21:05:28.399 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-12-25T21:05:28.399 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-25T21:05:28.442 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-25T21:05:28.442 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-12-25T21:05:28.442 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/usr/bin/daemon-helper 2023-12-25T21:05:28.470 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-25T21:05:28.538 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-12-25T21:05:28.539 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/daemon-helper 2023-12-25T21:05:28.569 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-25T21:05:28.639 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-25T21:05:28.639 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-12-25T21:05:28.640 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-25T21:05:28.668 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-25T21:05:28.736 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-12-25T21:05:28.736 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-25T21:05:28.766 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-25T21:05:28.836 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-25T21:05:28.836 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-12-25T21:05:28.836 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/usr/bin/stdin-killer 2023-12-25T21:05:28.863 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-25T21:05:28.931 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-12-25T21:05:28.932 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/stdin-killer 2023-12-25T21:05:28.960 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-25T21:05:29.030 INFO:teuthology.run_tasks:Running task print... 2023-12-25T21:05:29.043 INFO:teuthology.task.print:**** done install task... 2023-12-25T21:05:29.043 INFO:teuthology.run_tasks:Running task cephadm... 2023-12-25T21:05:29.134 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2'} 2023-12-25T21:05:29.134 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2023-12-25T21:05:29.135 INFO:tasks.cephadm:Cluster fsid is 5532d712-a369-11ee-95a5-87774f69a715 2023-12-25T21:05:29.135 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-12-25T21:05:29.135 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-12-25T21:05:29.135 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi061': '172.21.15.61', 'mon.smithi174': '172.21.15.174'} 2023-12-25T21:05:29.135 INFO:tasks.cephadm:Normalizing hostnames... 2023-12-25T21:05:29.135 DEBUG:teuthology.orchestra.run.smithi061:> sudo hostname $(hostname -s) 2023-12-25T21:05:29.164 DEBUG:teuthology.orchestra.run.smithi174:> sudo hostname $(hostname -s) 2023-12-25T21:05:29.194 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2023-12-25T21:05:29.195 DEBUG:teuthology.orchestra.run.smithi061:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-25T21:05:29.415 INFO:teuthology.orchestra.run.smithi061.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Dec 25 21:05 /home/ubuntu/cephtest/cephadm 2023-12-25T21:05:29.416 DEBUG:teuthology.orchestra.run.smithi174:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-25T21:05:29.565 INFO:teuthology.orchestra.run.smithi174.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Dec 25 21:05 /home/ubuntu/cephtest/cephadm 2023-12-25T21:05:29.566 DEBUG:teuthology.orchestra.run.smithi061:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-25T21:05:29.586 DEBUG:teuthology.orchestra.run.smithi174:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-25T21:05:29.618 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /etc/ceph 2023-12-25T21:05:29.654 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph 2023-12-25T21:05:29.684 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 777 /etc/ceph 2023-12-25T21:05:29.722 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 777 /etc/ceph 2023-12-25T21:05:29.753 INFO:tasks.cephadm:Writing seed config... 2023-12-25T21:05:29.754 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-12-25T21:05:29.754 INFO:tasks.cephadm: override: [client] debug client = 20 2023-12-25T21:05:29.755 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-12-25T21:05:29.755 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-12-25T21:05:29.755 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-12-25T21:05:29.755 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-12-25T21:05:29.755 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-12-25T21:05:29.755 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-12-25T21:05:29.755 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-12-25T21:05:29.755 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-12-25T21:05:29.755 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-12-25T21:05:29.755 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-12-25T21:05:29.756 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-12-25T21:05:29.756 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-12-25T21:05:29.756 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-12-25T21:05:29.756 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-12-25T21:05:29.756 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-12-25T21:05:29.756 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-12-25T21:05:29.756 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-12-25T21:05:29.756 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-12-25T21:05:29.756 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-12-25T21:05:29.756 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-12-25T21:05:29.756 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-12-25T21:05:29.756 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-12-25T21:05:29.756 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-12-25T21:05:29.757 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-12-25T21:05:29.758 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-12-25T21:05:29.758 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-12-25T21:05:29.758 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-12-25T21:05:29.758 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-12-25T21:05:29.758 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-12-25T21:05:29.758 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-12-25T21:05:29.758 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-12-25T21:05:29.758 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-12-25T21:05:29.758 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-12-25T21:05:29.758 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-12-25T21:05:29.758 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-12-25T21:05:29.758 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-12-25T21:05:29.759 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-12-25T21:05:29.759 DEBUG:teuthology.orchestra.run.smithi061:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-12-25T21:05:29.780 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 5532d712-a369-11ee-95a5-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-12-25T21:05:29.781 DEBUG:teuthology.orchestra.run.smithi061:mon.smithi061> sudo journalctl -f -n 0 -u ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi061.service 2023-12-25T21:05:29.824 INFO:tasks.cephadm:Bootstrapping... 2023-12-25T21:05:29.824 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 5532d712-a369-11ee-95a5-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.61 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-12-25T21:05:29.851 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:-- Logs begin at Mon 2023-12-25 20:53:30 UTC. -- 2023-12-25T21:05:30.100 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-12-25T21:05:30.985 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: 4.6.1 2023-12-25T21:05:30.992 INFO:teuthology.orchestra.run.smithi061.stderr:Verifying podman|docker is present... 2023-12-25T21:05:30.992 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-12-25T21:05:31.016 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: 4.6.1 2023-12-25T21:05:31.029 INFO:teuthology.orchestra.run.smithi061.stderr:Verifying lvm2 is present... 2023-12-25T21:05:31.029 INFO:teuthology.orchestra.run.smithi061.stderr:Verifying time synchronization is in place... 2023-12-25T21:05:31.029 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl is-enabled chrony.service 2023-12-25T21:05:31.037 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-12-25T21:05:31.038 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl is-active chrony.service 2023-12-25T21:05:31.044 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: inactive 2023-12-25T21:05:31.045 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl is-enabled chronyd.service 2023-12-25T21:05:31.051 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: enabled 2023-12-25T21:05:31.052 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl is-active chronyd.service 2023-12-25T21:05:31.058 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: active 2023-12-25T21:05:31.059 INFO:teuthology.orchestra.run.smithi061.stderr:Unit chronyd.service is enabled and running 2023-12-25T21:05:31.059 INFO:teuthology.orchestra.run.smithi061.stderr:Repeating the final host check... 2023-12-25T21:05:31.059 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-12-25T21:05:31.080 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: 4.6.1 2023-12-25T21:05:31.091 INFO:teuthology.orchestra.run.smithi061.stderr:podman|docker (/bin/podman) is present 2023-12-25T21:05:31.091 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl is present 2023-12-25T21:05:31.091 INFO:teuthology.orchestra.run.smithi061.stderr:lvcreate is present 2023-12-25T21:05:31.091 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl is-enabled chrony.service 2023-12-25T21:05:31.098 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-12-25T21:05:31.098 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl is-active chrony.service 2023-12-25T21:05:31.103 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: inactive 2023-12-25T21:05:31.104 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl is-enabled chronyd.service 2023-12-25T21:05:31.110 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: enabled 2023-12-25T21:05:31.110 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl is-active chronyd.service 2023-12-25T21:05:31.116 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: active 2023-12-25T21:05:31.117 INFO:teuthology.orchestra.run.smithi061.stderr:Unit chronyd.service is enabled and running 2023-12-25T21:05:31.117 INFO:teuthology.orchestra.run.smithi061.stderr:Host looks OK 2023-12-25T21:05:31.117 INFO:teuthology.orchestra.run.smithi061.stderr:Cluster fsid: 5532d712-a369-11ee-95a5-87774f69a715 2023-12-25T21:05:31.117 INFO:teuthology.orchestra.run.smithi061.stderr:Acquiring lock 139968974834152 on /run/cephadm/5532d712-a369-11ee-95a5-87774f69a715.lock 2023-12-25T21:05:31.118 INFO:teuthology.orchestra.run.smithi061.stderr:Lock 139968974834152 acquired on /run/cephadm/5532d712-a369-11ee-95a5-87774f69a715.lock 2023-12-25T21:05:31.118 INFO:teuthology.orchestra.run.smithi061.stderr:Verifying IP 172.21.15.61 port 3300 ... 2023-12-25T21:05:31.119 INFO:teuthology.orchestra.run.smithi061.stderr:Verifying IP 172.21.15.61 port 6789 ... 2023-12-25T21:05:31.119 INFO:teuthology.orchestra.run.smithi061.stderr:Base mon IP is 172.21.15.61, final addrv is [v2:172.21.15.61:3300,v1:172.21.15.61:6789] 2023-12-25T21:05:31.119 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /sbin/ip route ls 2023-12-25T21:05:31.121 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.61 metric 100 2023-12-25T21:05:31.121 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.61 metric 100 2023-12-25T21:05:31.122 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /sbin/ip -6 route ls 2023-12-25T21:05:31.124 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-12-25T21:05:31.124 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2023-12-25T21:05:31.124 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2023-12-25T21:05:31.125 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /sbin/ip -6 addr ls 2023-12-25T21:05:31.127 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-12-25T21:05:31.127 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-12-25T21:05:31.127 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-12-25T21:05:31.127 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-12-25T21:05:31.127 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: inet6 fe80::ec4:7aff:feb6:f8b9/64 scope link 2023-12-25T21:05:31.127 INFO:teuthology.orchestra.run.smithi061.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-12-25T21:05:31.129 INFO:teuthology.orchestra.run.smithi061.stderr:Mon IP 172.21.15.61 is in CIDR network 172.21.0.0/20 2023-12-25T21:05:31.129 INFO:teuthology.orchestra.run.smithi061.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-12-25T21:05:31.131 INFO:teuthology.orchestra.run.smithi061.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2023-12-25T21:05:31.131 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2023-12-25T21:05:31.230 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2023-12-25T21:05:37.778 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: Getting image source signatures 2023-12-25T21:05:37.811 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2023-12-25T21:05:37.811 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-12-25T21:05:51.549 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-12-25T21:05:51.598 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: Writing manifest to image destination 2023-12-25T21:05:52.370 INFO:teuthology.orchestra.run.smithi061.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-12-25T21:05:52.381 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2023-12-25T21:05:52.959 INFO:teuthology.orchestra.run.smithi061.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-12-25T21:05:53.311 INFO:teuthology.orchestra.run.smithi061.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-12-25T21:05:53.311 INFO:teuthology.orchestra.run.smithi061.stderr:Extracting ceph user uid/gid from container image... 2023-12-25T21:05:53.311 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2023-12-25T21:05:53.556 INFO:teuthology.orchestra.run.smithi061.stderr:stat: 167 167 2023-12-25T21:05:53.892 INFO:teuthology.orchestra.run.smithi061.stderr:Creating initial keys... 2023-12-25T21:05:53.892 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-12-25T21:05:54.173 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-authtool: AQCy7ollM1RZCRAAEH8hA0/WTEyEk30WAL0QbA== 2023-12-25T21:05:54.522 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-12-25T21:05:54.790 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-authtool: AQCy7oll/v1ULhAA5y9vfbQ9xVWy8QtAFdlHlQ== 2023-12-25T21:05:55.135 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-12-25T21:05:55.414 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-authtool: AQCz7oll2DnoFxAACbxxqXZBUbaP6r9H/ZRbUg== 2023-12-25T21:05:55.753 INFO:teuthology.orchestra.run.smithi061.stderr:Creating initial monmap... 2023-12-25T21:05:55.753 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpo0pnf_rf:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 5532d712-a369-11ee-95a5-87774f69a715 --addv smithi061 [v2:172.21.15.61:3300,v1:172.21.15.61:6789] /tmp/monmap 2023-12-25T21:05:56.031 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-25T21:05:56.032 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 5532d712-a369-11ee-95a5-87774f69a715 2023-12-25T21:05:56.032 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-25T21:05:56.366 INFO:teuthology.orchestra.run.smithi061.stderr:monmaptool for smithi061 [v2:172.21.15.61:3300,v1:172.21.15.61:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-25T21:05:56.366 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/monmaptool: set fsid to 5532d712-a369-11ee-95a5-87774f69a715 2023-12-25T21:05:56.366 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-25T21:05:56.366 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-12-25T21:05:56.366 INFO:teuthology.orchestra.run.smithi061.stderr:Creating mon... 2023-12-25T21:05:56.367 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/mon.smithi061:/var/lib/ceph/mon/ceph-smithi061:z -v /tmp/ceph-tmp6swqf_fn:/tmp/keyring:z -v /tmp/ceph-tmpo0pnf_rf:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi061 --fsid 5532d712-a369-11ee-95a5-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-12-25T21:05:56.650 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.617+0000 7fd8b31b3700 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-25T21:05:56.650 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 1 imported monmap: 2023-12-25T21:05:56.651 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: epoch 0 2023-12-25T21:05:56.651 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: fsid 5532d712-a369-11ee-95a5-87774f69a715 2023-12-25T21:05:56.651 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: last_changed 2023-12-25T21:05:56.011996+0000 2023-12-25T21:05:56.651 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: created 2023-12-25T21:05:56.011996+0000 2023-12-25T21:05:56.651 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-12-25T21:05:56.652 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-12-25T21:05:56.652 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] mon.smithi061 2023-12-25T21:05:56.652 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-12-25T21:05:56.652 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 0 /usr/bin/ceph-mon: set fsid to 5532d712-a369-11ee-95a5-87774f69a715 2023-12-25T21:05:56.652 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: RocksDB version: 6.8.1 2023-12-25T21:05:56.652 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-12-25T21:05:56.653 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-25T21:05:56.653 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Compile date May 13 2021 2023-12-25T21:05:56.653 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: DB SUMMARY 2023-12-25T21:05:56.653 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-12-25T21:05:56.653 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi061/store.db dir, Total Num: 0, files: 2023-12-25T21:05:56.653 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-12-25T21:05:56.653 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi061/store.db: 2023-12-25T21:05:56.653 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-12-25T21:05:56.653 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.error_if_exists: 0 2023-12-25T21:05:56.654 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.create_if_missing: 1 2023-12-25T21:05:56.654 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.paranoid_checks: 1 2023-12-25T21:05:56.654 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.env: 0x5623253841c0 2023-12-25T21:05:56.654 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.fs: Posix File System 2023-12-25T21:05:56.654 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.info_log: 0x562326232ec0 2023-12-25T21:05:56.654 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-25T21:05:56.654 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.statistics: (nil) 2023-12-25T21:05:56.654 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.use_fsync: 0 2023-12-25T21:05:56.654 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.max_log_file_size: 0 2023-12-25T21:05:56.654 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-25T21:05:56.654 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-25T21:05:56.654 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-25T21:05:56.655 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-25T21:05:56.655 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.allow_fallocate: 1 2023-12-25T21:05:56.655 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-25T21:05:56.655 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-25T21:05:56.655 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.use_direct_reads: 0 2023-12-25T21:05:56.655 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-25T21:05:56.655 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.create_missing_column_families: 0 2023-12-25T21:05:56.655 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.db_log_dir: 2023-12-25T21:05:56.655 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi061/store.db 2023-12-25T21:05:56.655 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-25T21:05:56.656 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.max_subcompactions: 1 2023-12-25T21:05:56.656 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.max_background_flushes: -1 2023-12-25T21:05:56.656 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-25T21:05:56.656 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-25T21:05:56.656 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-25T21:05:56.656 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-25T21:05:56.657 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-25T21:05:56.657 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.advise_random_on_open: 1 2023-12-25T21:05:56.657 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-25T21:05:56.657 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.write_buffer_manager: 0x562326239530 2023-12-25T21:05:56.657 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-25T21:05:56.657 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-25T21:05:56.658 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-25T21:05:56.658 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-25T21:05:56.658 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.rate_limiter: (nil) 2023-12-25T21:05:56.658 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-25T21:05:56.658 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-25T21:05:56.658 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-25T21:05:56.658 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-25T21:05:56.658 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.unordered_write: 0 2023-12-25T21:05:56.658 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-25T21:05:56.658 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-25T21:05:56.658 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-25T21:05:56.658 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-25T21:05:56.658 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.row_cache: None 2023-12-25T21:05:56.658 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.wal_filter: None 2023-12-25T21:05:56.659 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-25T21:05:56.659 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-25T21:05:56.659 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.preserve_deletes: 0 2023-12-25T21:05:56.659 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.two_write_queues: 0 2023-12-25T21:05:56.659 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.manual_wal_flush: 0 2023-12-25T21:05:56.659 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.atomic_flush: 0 2023-12-25T21:05:56.659 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-25T21:05:56.659 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-25T21:05:56.659 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-25T21:05:56.659 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.log_readahead_size: 0 2023-12-25T21:05:56.659 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-12-25T21:05:56.659 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.max_background_jobs: 2 2023-12-25T21:05:56.660 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.max_background_compactions: -1 2023-12-25T21:05:56.660 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-25T21:05:56.660 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-25T21:05:56.660 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-25T21:05:56.660 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.max_total_wal_size: 0 2023-12-25T21:05:56.660 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-25T21:05:56.660 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-25T21:05:56.660 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-25T21:05:56.660 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-25T21:05:56.660 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.max_open_files: -1 2023-12-25T21:05:56.661 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.bytes_per_sync: 0 2023-12-25T21:05:56.661 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-25T21:05:56.661 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-25T21:05:56.661 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-25T21:05:56.661 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Compression algorithms supported: 2023-12-25T21:05:56.661 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-25T21:05:56.661 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: kZSTD supported: 0 2023-12-25T21:05:56.661 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: kXpressCompression supported: 0 2023-12-25T21:05:56.661 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-25T21:05:56.661 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: kLZ4Compression supported: 1 2023-12-25T21:05:56.662 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: kBZip2Compression supported: 0 2023-12-25T21:05:56.662 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: kZlibCompression supported: 1 2023-12-25T21:05:56.662 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: kSnappyCompression supported: 1 2023-12-25T21:05:56.662 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.618+0000 7fd8b31b3700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-25T21:05:56.662 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-12-25T21:05:56.662 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-12-25T21:05:56.663 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi061/store.db/MANIFEST-000001 2023-12-25T21:05:56.663 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-12-25T21:05:56.663 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-12-25T21:05:56.663 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-12-25T21:05:56.663 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-25T21:05:56.663 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.merge_operator: 2023-12-25T21:05:56.663 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.compaction_filter: None 2023-12-25T21:05:56.663 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.compaction_filter_factory: None 2023-12-25T21:05:56.663 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-25T21:05:56.663 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-25T21:05:56.663 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56232614a120) 2023-12-25T21:05:56.663 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-12-25T21:05:56.663 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-25T21:05:56.664 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-25T21:05:56.664 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-12-25T21:05:56.664 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: index_type: 0 2023-12-25T21:05:56.664 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-12-25T21:05:56.664 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-12-25T21:05:56.664 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-12-25T21:05:56.664 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-12-25T21:05:56.664 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: checksum: 1 2023-12-25T21:05:56.664 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-12-25T21:05:56.664 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: block_cache: 0x562326180d10 2023-12-25T21:05:56.665 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-12-25T21:05:56.665 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-12-25T21:05:56.665 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-12-25T21:05:56.665 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-12-25T21:05:56.665 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-12-25T21:05:56.666 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-12-25T21:05:56.666 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-12-25T21:05:56.666 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-12-25T21:05:56.666 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-12-25T21:05:56.666 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-12-25T21:05:56.666 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-12-25T21:05:56.666 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-12-25T21:05:56.666 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-12-25T21:05:56.667 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-12-25T21:05:56.667 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-12-25T21:05:56.667 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-25T21:05:56.667 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-12-25T21:05:56.667 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-12-25T21:05:56.667 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-12-25T21:05:56.667 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: format_version: 2 2023-12-25T21:05:56.667 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-12-25T21:05:56.667 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: block_align: 0 2023-12-25T21:05:56.667 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-12-25T21:05:56.667 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-25T21:05:56.667 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-25T21:05:56.667 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.compression: NoCompression 2023-12-25T21:05:56.667 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-25T21:05:56.668 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-25T21:05:56.668 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-25T21:05:56.668 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.num_levels: 7 2023-12-25T21:05:56.668 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-25T21:05:56.668 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-25T21:05:56.668 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-25T21:05:56.668 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-25T21:05:56.668 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-25T21:05:56.669 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-25T21:05:56.669 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-25T21:05:56.669 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-25T21:05:56.669 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-25T21:05:56.669 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-25T21:05:56.669 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.compression_opts.level: 32767 2023-12-25T21:05:56.669 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-25T21:05:56.669 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-25T21:05:56.669 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-25T21:05:56.669 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.compression_opts.enabled: false 2023-12-25T21:05:56.669 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-25T21:05:56.669 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-25T21:05:56.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-25T21:05:56.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-25T21:05:56.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-25T21:05:56.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-25T21:05:56.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-25T21:05:56.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-25T21:05:56.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-25T21:05:56.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-25T21:05:56.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-25T21:05:56.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-25T21:05:56.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-25T21:05:56.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-25T21:05:56.670 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-25T21:05:56.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-25T21:05:56.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-25T21:05:56.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.arena_block_size: 4194304 2023-12-25T21:05:56.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-25T21:05:56.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-25T21:05:56.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-25T21:05:56.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-25T21:05:56.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-25T21:05:56.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-25T21:05:56.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-25T21:05:56.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-25T21:05:56.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-25T21:05:56.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-25T21:05:56.671 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-25T21:05:56.672 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-25T21:05:56.672 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-25T21:05:56.672 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-25T21:05:56.672 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.table_properties_collectors: 2023-12-25T21:05:56.672 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.inplace_update_support: 0 2023-12-25T21:05:56.672 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-25T21:05:56.672 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-25T21:05:56.672 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-25T21:05:56.672 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-25T21:05:56.672 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.bloom_locality: 0 2023-12-25T21:05:56.672 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.max_successive_merges: 0 2023-12-25T21:05:56.673 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-25T21:05:56.673 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-25T21:05:56.673 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.force_consistency_checks: 0 2023-12-25T21:05:56.673 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-25T21:05:56.673 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.ttl: 2592000 2023-12-25T21:05:56.673 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.619+0000 7fd8b31b3700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-25T21:05:56.673 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.620+0000 7fd8b31b3700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi061/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-25T21:05:56.673 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-12-25T21:05:56.674 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.620+0000 7fd8b31b3700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-12-25T21:05:56.674 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-12-25T21:05:56.674 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.620+0000 7fd8b31b3700 4 rocksdb: DB pointer 0x562326245800 2023-12-25T21:05:56.674 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.620+0000 7fd89c04c700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-12-25T21:05:56.674 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.620+0000 7fd89c04c700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-12-25T21:05:56.674 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-12-25T21:05:56.674 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-25T21:05:56.674 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-25T21:05:56.674 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-25T21:05:56.674 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-25T21:05:56.674 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-25T21:05:56.674 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-25T21:05:56.674 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-25T21:05:56.675 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-12-25T21:05:56.675 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-25T21:05:56.675 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-25T21:05:56.675 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-25T21:05:56.675 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-25T21:05:56.675 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-25T21:05:56.675 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-12-25T21:05:56.675 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-25T21:05:56.675 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-25T21:05:56.675 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-25T21:05:56.675 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-25T21:05:56.675 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-12-25T21:05:56.675 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-25T21:05:56.675 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-12-25T21:05:56.675 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-12-25T21:05:56.676 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-12-25T21:05:56.676 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-25T21:05:56.676 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-25T21:05:56.676 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-25T21:05:56.676 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-12-25T21:05:56.676 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-12-25T21:05:56.676 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-12-25T21:05:56.676 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-25T21:05:56.676 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-25T21:05:56.676 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-25T21:05:56.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-25T21:05:56.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-25T21:05:56.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-12-25T21:05:56.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-25T21:05:56.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-25T21:05:56.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-25T21:05:56.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-25T21:05:56.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-12-25T21:05:56.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-25T21:05:56.677 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-12-25T21:05:56.678 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-12-25T21:05:56.678 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-12-25T21:05:56.678 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-25T21:05:56.678 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-25T21:05:56.678 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-25T21:05:56.678 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-12-25T21:05:56.678 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-12-25T21:05:56.678 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: 2023-12-25T21:05:56.678 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.622+0000 7fd8b31b3700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-12-25T21:05:56.678 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.622+0000 7fd8b31b3700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-12-25T21:05:56.678 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph-mon: debug 2023-12-25T21:05:56.622+0000 7fd8b31b3700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi061 for mon.smithi061 2023-12-25T21:05:56.966 INFO:teuthology.orchestra.run.smithi061.stderr:create mon.smithi061 on 2023-12-25T21:05:56.966 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/5532d712-a369-11ee-95a5-87774f69a715 2023-12-25T21:05:56.971 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl disable ceph.target 2023-12-25T21:05:56.978 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-12-25T21:05:57.069 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl enable ceph.target 2023-12-25T21:05:57.078 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-12-25T21:05:57.162 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl start ceph.target 2023-12-25T21:05:57.170 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl enable ceph-5532d712-a369-11ee-95a5-87774f69a715.target 2023-12-25T21:05:57.176 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-5532d712-a369-11ee-95a5-87774f69a715.target → /etc/systemd/system/ceph-5532d712-a369-11ee-95a5-87774f69a715.target. 2023-12-25T21:05:57.176 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-5532d712-a369-11ee-95a5-87774f69a715.target → /etc/systemd/system/ceph-5532d712-a369-11ee-95a5-87774f69a715.target. 2023-12-25T21:05:57.258 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl start ceph-5532d712-a369-11ee-95a5-87774f69a715.target 2023-12-25T21:05:57.265 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl daemon-reload 2023-12-25T21:05:57.358 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl stop ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi061 2023-12-25T21:05:57.395 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl reset-failed ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi061 2023-12-25T21:05:57.400 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Failed to reset failed state of unit ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi061.service: Unit ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi061.service not loaded. 2023-12-25T21:05:57.401 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl enable ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi061 2023-12-25T21:05:57.407 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Created symlink /etc/systemd/system/ceph-5532d712-a369-11ee-95a5-87774f69a715.target.wants/ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi061.service → /etc/systemd/system/ceph-5532d712-a369-11ee-95a5-87774f69a715@.service. 2023-12-25T21:05:57.496 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl start ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi061 2023-12-25T21:05:57.539 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:05:57 smithi061 systemd[1]: Starting Ceph mon.smithi061 for 5532d712-a369-11ee-95a5-87774f69a715... 2023-12-25T21:05:57.819 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-25T21:05:57.827 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: disabled 2023-12-25T21:05:57.828 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl is-active firewalld.service 2023-12-25T21:05:57.835 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: inactive 2023-12-25T21:05:57.836 INFO:teuthology.orchestra.run.smithi061.stderr:firewalld.service is not enabled 2023-12-25T21:05:57.837 INFO:teuthology.orchestra.run.smithi061.stderr:Not possible to enable service . firewalld.service is not available 2023-12-25T21:05:57.837 INFO:teuthology.orchestra.run.smithi061.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-12-25T21:05:57.837 INFO:teuthology.orchestra.run.smithi061.stderr:Waiting for mon to start... 2023-12-25T21:05:57.837 INFO:teuthology.orchestra.run.smithi061.stderr:Waiting for mon... 2023-12-25T21:05:57.837 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/mon.smithi061:/var/lib/ceph/mon/ceph-smithi061:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2023-12-25T21:05:58.205 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:05:57 smithi061 bash[111296]: 5db505c0fe136512097e58f914dd21645ff3f0a9171515118b6e137b991fa0af 2023-12-25T21:05:58.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:05:57 smithi061 systemd[1]: Started Ceph mon.smithi061 for 5532d712-a369-11ee-95a5-87774f69a715. 2023-12-25T21:05:58.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:05:57 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111311]: cluster 2023-12-25T21:05:57.844803+0000 mon.smithi061 (mon.0) 0 : [INF] mkfs 5532d712-a369-11ee-95a5-87774f69a715 2023-12-25T21:05:58.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:05:57 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111311]: cluster 2023-12-25T21:05:57.826849+0000 mon.smithi061 (mon.0) 1 : cluster [INF] mon.smithi061 is new leader, mons smithi061 in quorum (ranks 0) 2023-12-25T21:05:58.241 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.238+0000 7f7ad5b6b700 1 Processor -- start 2023-12-25T21:05:58.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.238+0000 7f7ad5b6b700 1 -- start start 2023-12-25T21:05:58.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.238+0000 7f7ad5b6b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ad00fab80 0x7f7ad00fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:05:58.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.238+0000 7f7ad5b6b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad00fb490 con 0x7f7ad00fab80 2023-12-25T21:05:58.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.239+0000 7f7acf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ad00fab80 0x7f7ad00fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:05:58.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.239+0000 7f7acf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ad00fab80 0x7f7ad00fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:38174/0 (socket says 172.21.15.61:38174) 2023-12-25T21:05:58.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.239+0000 7f7acf7fe700 1 -- 172.21.15.61:0/306725349 learned_addr learned my addr 172.21.15.61:0/306725349 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:05:58.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.239+0000 7f7acf7fe700 1 -- 172.21.15.61:0/306725349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad00fb5d0 con 0x7f7ad00fab80 2023-12-25T21:05:58.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.239+0000 7f7acf7fe700 1 --2- 172.21.15.61:0/306725349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ad00fab80 0x7f7ad00fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f7ac000b5f0 tx=0x7f7ac0006ee0).ready entity=mon.0 client_cookie=f6e9a0d46cd90a55 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:05:58.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.240+0000 7f7ace7fc700 1 -- 172.21.15.61:0/306725349 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ac000d600 con 0x7f7ad00fab80 2023-12-25T21:05:58.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.240+0000 7f7ace7fc700 1 -- 172.21.15.61:0/306725349 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac000d760 con 0x7f7ad00fab80 2023-12-25T21:05:58.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.240+0000 7f7ace7fc700 1 -- 172.21.15.61:0/306725349 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ac000da30 con 0x7f7ad00fab80 2023-12-25T21:05:58.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.240+0000 7f7ad5b6b700 1 -- 172.21.15.61:0/306725349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ad00fab80 msgr2=0x7f7ad00fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:05:58.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.240+0000 7f7ad5b6b700 1 --2- 172.21.15.61:0/306725349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ad00fab80 0x7f7ad00fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f7ac000b5f0 tx=0x7f7ac0006ee0).stop 2023-12-25T21:05:58.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.240+0000 7f7ad5b6b700 1 -- 172.21.15.61:0/306725349 shutdown_connections 2023-12-25T21:05:58.243 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.240+0000 7f7ad5b6b700 1 --2- 172.21.15.61:0/306725349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ad00fab80 0x7f7ad00fafa0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:05:58.243 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.240+0000 7f7ad5b6b700 1 -- 172.21.15.61:0/306725349 >> 172.21.15.61:0/306725349 conn(0x7f7ad00f63c0 msgr2=0x7f7ad00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:05:58.243 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.240+0000 7f7ad5b6b700 1 -- 172.21.15.61:0/306725349 shutdown_connections 2023-12-25T21:05:58.243 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.240+0000 7f7ad5b6b700 1 -- 172.21.15.61:0/306725349 wait complete. 2023-12-25T21:05:58.243 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.241+0000 7f7ad5b6b700 1 Processor -- start 2023-12-25T21:05:58.243 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.241+0000 7f7ad5b6b700 1 -- start start 2023-12-25T21:05:58.244 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.241+0000 7f7ad5b6b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ad00fab80 0x7f7ad010cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:05:58.244 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.241+0000 7f7ad5b6b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad0070470 con 0x7f7ad00fab80 2023-12-25T21:05:58.244 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.242+0000 7f7acf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ad00fab80 0x7f7ad010cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:05:58.244 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.242+0000 7f7acf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ad00fab80 0x7f7ad010cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:38184/0 (socket says 172.21.15.61:38184) 2023-12-25T21:05:58.244 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.242+0000 7f7acf7fe700 1 -- 172.21.15.61:0/3436995604 learned_addr learned my addr 172.21.15.61:0/3436995604 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:05:58.245 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.242+0000 7f7acf7fe700 1 -- 172.21.15.61:0/3436995604 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac000b200 con 0x7f7ad00fab80 2023-12-25T21:05:58.245 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.242+0000 7f7acf7fe700 1 --2- 172.21.15.61:0/3436995604 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ad00fab80 0x7f7ad010cf50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f7ac0006c70 tx=0x7f7ac0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:05:58.245 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.242+0000 7f7ac7fff700 1 -- 172.21.15.61:0/3436995604 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ac0011b40 con 0x7f7ad00fab80 2023-12-25T21:05:58.245 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.242+0000 7f7ac7fff700 1 -- 172.21.15.61:0/3436995604 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac0011ca0 con 0x7f7ad00fab80 2023-12-25T21:05:58.245 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.242+0000 7f7ad5b6b700 1 -- 172.21.15.61:0/3436995604 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad0070670 con 0x7f7ad00fab80 2023-12-25T21:05:58.245 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.242+0000 7f7ac7fff700 1 -- 172.21.15.61:0/3436995604 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ac001d430 con 0x7f7ad00fab80 2023-12-25T21:05:58.245 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.242+0000 7f7ad5b6b700 1 -- 172.21.15.61:0/3436995604 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad010d600 con 0x7f7ad00fab80 2023-12-25T21:05:58.245 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.243+0000 7f7ac7fff700 1 -- 172.21.15.61:0/3436995604 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f7ac000f070 con 0x7f7ad00fab80 2023-12-25T21:05:58.245 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.243+0000 7f7ac7fff700 1 -- 172.21.15.61:0/3436995604 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f7ac0027460 con 0x7f7ad00fab80 2023-12-25T21:05:58.245 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.243+0000 7f7ad5b6b700 1 -- 172.21.15.61:0/3436995604 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad0045ad0 con 0x7f7ad00fab80 2023-12-25T21:05:58.246 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.246+0000 7f7ac7fff700 1 -- 172.21.15.61:0/3436995604 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f7ac0016070 con 0x7f7ad00fab80 2023-12-25T21:05:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.315+0000 7f7ad5b6b700 1 -- 172.21.15.61:0/3436995604 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f7ad00fbec0 con 0x7f7ad00fab80 2023-12-25T21:05:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.315+0000 7f7ac7fff700 1 -- 172.21.15.61:0/3436995604 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f7ac0035160 con 0x7f7ad00fab80 2023-12-25T21:05:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: cluster: 2023-12-25T21:05:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: id: 5532d712-a369-11ee-95a5-87774f69a715 2023-12-25T21:05:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-12-25T21:05:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: services: 2023-12-25T21:05:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi061 (age 0.473049s) 2023-12-25T21:05:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mgr: no daemons active 2023-12-25T21:05:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-12-25T21:05:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: data: 2023-12-25T21:05:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-12-25T21:05:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-12-25T21:05:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-12-25T21:05:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: pgs: 2023-12-25T21:05:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.316+0000 7f7ad5b6b700 1 -- 172.21.15.61:0/3436995604 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ad00fab80 msgr2=0x7f7ad010cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:05:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.316+0000 7f7ad5b6b700 1 --2- 172.21.15.61:0/3436995604 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ad00fab80 0x7f7ad010cf50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f7ac0006c70 tx=0x7f7ac0016000).stop 2023-12-25T21:05:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.316+0000 7f7ad5b6b700 1 -- 172.21.15.61:0/3436995604 shutdown_connections 2023-12-25T21:05:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.316+0000 7f7ad5b6b700 1 --2- 172.21.15.61:0/3436995604 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ad00fab80 0x7f7ad010cf50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:05:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.316+0000 7f7ad5b6b700 1 -- 172.21.15.61:0/3436995604 >> 172.21.15.61:0/3436995604 conn(0x7f7ad00f63c0 msgr2=0x7f7ad00fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:05:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.316+0000 7f7ad5b6b700 1 -- 172.21.15.61:0/3436995604 shutdown_connections 2023-12-25T21:05:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:58.316+0000 7f7ad5b6b700 1 -- 172.21.15.61:0/3436995604 wait complete. 2023-12-25T21:05:58.667 INFO:teuthology.orchestra.run.smithi061.stderr:mon is available 2023-12-25T21:05:58.667 INFO:teuthology.orchestra.run.smithi061.stderr:Assimilating anything we can from ceph.conf... 2023-12-25T21:05:58.667 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z -v /var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/mon.smithi061:/var/lib/ceph/mon/ceph-smithi061:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi061/config 2023-12-25T21:05:58.864 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:05:58 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111311]: cluster 2023-12-25T21:05:57.843465+0000 mon.smithi061 (mon.0) 2 : cluster [INF] mon.smithi061 is new leader, mons smithi061 in quorum (ranks 0) 2023-12-25T21:05:58.864 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:05:58 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111311]: cluster 2023-12-25T21:05:57.843707+0000 mon.smithi061 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi061=[v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0]} 2023-12-25T21:05:58.864 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:05:58 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111311]: cluster 2023-12-25T21:05:57.847071+0000 mon.smithi061 (mon.0) 4 : cluster [DBG] fsmap 2023-12-25T21:05:58.864 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:05:58 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111311]: cluster 2023-12-25T21:05:57.849067+0000 mon.smithi061 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-12-25T21:05:58.864 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:05:58 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111311]: cluster 2023-12-25T21:05:57.849470+0000 mon.smithi061 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-12-25T21:05:58.864 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:05:58 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111311]: audit 2023-12-25T21:05:58.316482+0000 mon.smithi061 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.61:0/3436995604' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-12-25T21:05:59.099 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.098+0000 7f83e191d700 1 Processor -- start 2023-12-25T21:05:59.101 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.098+0000 7f83e191d700 1 -- start start 2023-12-25T21:05:59.101 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.098+0000 7f83e191d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83dc070440 0x7f83dc0fe950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:05:59.101 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.098+0000 7f83e191d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83dc0fee40 con 0x7f83dc070440 2023-12-25T21:05:59.101 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.099+0000 7f83daffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83dc070440 0x7f83dc0fe950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:05:59.101 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.099+0000 7f83daffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83dc070440 0x7f83dc0fe950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:38196/0 (socket says 172.21.15.61:38196) 2023-12-25T21:05:59.101 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.099+0000 7f83daffd700 1 -- 172.21.15.61:0/3725711713 learned_addr learned my addr 172.21.15.61:0/3725711713 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:05:59.101 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.099+0000 7f83daffd700 1 -- 172.21.15.61:0/3725711713 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83dc0fef80 con 0x7f83dc070440 2023-12-25T21:05:59.101 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.099+0000 7f83daffd700 1 --2- 172.21.15.61:0/3725711713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83dc070440 0x7f83dc0fe950 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f83d000b5f0 tx=0x7f83d0006ee0).ready entity=mon.0 client_cookie=73b7bfedfa5603bd server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:05:59.104 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.099+0000 7f83d9ffb700 1 -- 172.21.15.61:0/3725711713 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83d000d600 con 0x7f83dc070440 2023-12-25T21:05:59.104 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.099+0000 7f83d9ffb700 1 -- 172.21.15.61:0/3725711713 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d000d760 con 0x7f83dc070440 2023-12-25T21:05:59.104 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.099+0000 7f83d9ffb700 1 -- 172.21.15.61:0/3725711713 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83d000da30 con 0x7f83dc070440 2023-12-25T21:05:59.105 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.100+0000 7f83e191d700 1 -- 172.21.15.61:0/3725711713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83dc070440 msgr2=0x7f83dc0fe950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:05:59.105 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.100+0000 7f83e191d700 1 --2- 172.21.15.61:0/3725711713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83dc070440 0x7f83dc0fe950 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f83d000b5f0 tx=0x7f83d0006ee0).stop 2023-12-25T21:05:59.105 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.100+0000 7f83e191d700 1 -- 172.21.15.61:0/3725711713 shutdown_connections 2023-12-25T21:05:59.105 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.100+0000 7f83e191d700 1 --2- 172.21.15.61:0/3725711713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83dc070440 0x7f83dc0fe950 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:05:59.105 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.100+0000 7f83e191d700 1 -- 172.21.15.61:0/3725711713 >> 172.21.15.61:0/3725711713 conn(0x7f83dc0f63b0 msgr2=0x7f83dc0f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:05:59.105 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.100+0000 7f83e191d700 1 -- 172.21.15.61:0/3725711713 shutdown_connections 2023-12-25T21:05:59.105 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.100+0000 7f83e191d700 1 -- 172.21.15.61:0/3725711713 wait complete. 2023-12-25T21:05:59.105 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.101+0000 7f83e191d700 1 Processor -- start 2023-12-25T21:05:59.105 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.101+0000 7f83e191d700 1 -- start start 2023-12-25T21:05:59.105 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.101+0000 7f83e191d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83dc070440 0x7f83dc10a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:05:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.101+0000 7f83e191d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83dc10a740 con 0x7f83dc070440 2023-12-25T21:05:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.101+0000 7f83daffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83dc070440 0x7f83dc10a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:05:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.101+0000 7f83daffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83dc070440 0x7f83dc10a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:38208/0 (socket says 172.21.15.61:38208) 2023-12-25T21:05:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.101+0000 7f83daffd700 1 -- 172.21.15.61:0/611502147 learned_addr learned my addr 172.21.15.61:0/611502147 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:05:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.101+0000 7f83daffd700 1 -- 172.21.15.61:0/611502147 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d000b200 con 0x7f83dc070440 2023-12-25T21:05:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.101+0000 7f83daffd700 1 --2- 172.21.15.61:0/611502147 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83dc070440 0x7f83dc10a250 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f83d0006c70 tx=0x7f83d0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:05:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.102+0000 7f83c37fe700 1 -- 172.21.15.61:0/611502147 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83d0011b40 con 0x7f83dc070440 2023-12-25T21:05:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.102+0000 7f83c37fe700 1 -- 172.21.15.61:0/611502147 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d0011ca0 con 0x7f83dc070440 2023-12-25T21:05:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.102+0000 7f83e191d700 1 -- 172.21.15.61:0/611502147 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83dc10a940 con 0x7f83dc070440 2023-12-25T21:05:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.102+0000 7f83c37fe700 1 -- 172.21.15.61:0/611502147 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83d001d430 con 0x7f83dc070440 2023-12-25T21:05:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.102+0000 7f83e191d700 1 -- 172.21.15.61:0/611502147 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83dc10ad50 con 0x7f83dc070440 2023-12-25T21:05:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.102+0000 7f83c37fe700 1 -- 172.21.15.61:0/611502147 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f83d000f070 con 0x7f83dc070440 2023-12-25T21:05:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.102+0000 7f83c37fe700 1 -- 172.21.15.61:0/611502147 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f83d0027460 con 0x7f83dc070440 2023-12-25T21:05:59.106 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.103+0000 7f83e191d700 1 -- 172.21.15.61:0/611502147 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83dc045ad0 con 0x7f83dc070440 2023-12-25T21:05:59.107 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.105+0000 7f83c37fe700 1 -- 172.21.15.61:0/611502147 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f83d0016070 con 0x7f83dc070440 2023-12-25T21:05:59.172 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.172+0000 7f83e191d700 1 -- 172.21.15.61:0/611502147 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f83dc0ff170 con 0x7f83dc070440 2023-12-25T21:05:59.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.175+0000 7f83c37fe700 1 -- 172.21.15.61:0/611502147 <== mon.0 v2:172.21.15.61:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f83d001d5e0 con 0x7f83dc070440 2023-12-25T21:05:59.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.175+0000 7f83c37fe700 1 -- 172.21.15.61:0/611502147 <== mon.0 v2:172.21.15.61:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7f83d0036020 con 0x7f83dc070440 2023-12-25T21:05:59.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: [global] 2023-12-25T21:05:59.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: fsid = 5532d712-a369-11ee-95a5-87774f69a715 2023-12-25T21:05:59.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.61:3300,v1:172.21.15.61:6789] 2023-12-25T21:05:59.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-12-25T21:05:59.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-12-25T21:05:59.179 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-12-25T21:05:59.179 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-12-25T21:05:59.179 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.179 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: [mgr] 2023-12-25T21:05:59.179 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-12-25T21:05:59.179 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.179 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: [osd] 2023-12-25T21:05:59.179 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-12-25T21:05:59.179 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-12-25T21:05:59.179 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-12-25T21:05:59.179 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-12-25T21:05:59.179 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-12-25T21:05:59.179 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-12-25T21:05:59.179 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.176+0000 7f83e191d700 1 -- 172.21.15.61:0/611502147 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83dc070440 msgr2=0x7f83dc10a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:05:59.180 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.176+0000 7f83e191d700 1 --2- 172.21.15.61:0/611502147 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83dc070440 0x7f83dc10a250 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f83d0006c70 tx=0x7f83d0016000).stop 2023-12-25T21:05:59.180 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.176+0000 7f83e191d700 1 -- 172.21.15.61:0/611502147 shutdown_connections 2023-12-25T21:05:59.180 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.176+0000 7f83e191d700 1 --2- 172.21.15.61:0/611502147 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83dc070440 0x7f83dc10a250 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:05:59.180 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.176+0000 7f83e191d700 1 -- 172.21.15.61:0/611502147 >> 172.21.15.61:0/611502147 conn(0x7f83dc0f63b0 msgr2=0x7f83dc0f80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:05:59.180 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.176+0000 7f83e191d700 1 -- 172.21.15.61:0/611502147 shutdown_connections 2023-12-25T21:05:59.180 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.176+0000 7f83e191d700 1 -- 172.21.15.61:0/611502147 wait complete. 2023-12-25T21:05:59.558 INFO:teuthology.orchestra.run.smithi061.stderr:Generating new minimal ceph.conf... 2023-12-25T21:05:59.558 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z -v /var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/mon.smithi061:/var/lib/ceph/mon/ceph-smithi061:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi061/config 2023-12-25T21:05:59.984 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.979+0000 7fb05c85a700 1 Processor -- start 2023-12-25T21:05:59.984 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.979+0000 7fb05c85a700 1 -- start start 2023-12-25T21:05:59.985 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.980+0000 7fb05c85a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb058070350 0x7fb0580fe9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:05:59.985 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.980+0000 7fb05c85a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb058070770 con 0x7fb058070350 2023-12-25T21:05:59.985 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.980+0000 7fb05659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb058070350 0x7fb0580fe9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:05:59.986 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.980+0000 7fb05659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb058070350 0x7fb0580fe9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:39750/0 (socket says 172.21.15.61:39750) 2023-12-25T21:05:59.986 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.980+0000 7fb05659c700 1 -- 172.21.15.61:0/3977253343 learned_addr learned my addr 172.21.15.61:0/3977253343 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:05:59.986 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.980+0000 7fb05659c700 1 -- 172.21.15.61:0/3977253343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0580feeb0 con 0x7fb058070350 2023-12-25T21:05:59.986 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.980+0000 7fb05659c700 1 --2- 172.21.15.61:0/3977253343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb058070350 0x7fb0580fe9c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fb04800b5f0 tx=0x7fb048006ee0).ready entity=mon.0 client_cookie=3925c5f5d938d907 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:05:59.987 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.981+0000 7fb05559a700 1 -- 172.21.15.61:0/3977253343 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb04800d600 con 0x7fb058070350 2023-12-25T21:05:59.987 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.981+0000 7fb05559a700 1 -- 172.21.15.61:0/3977253343 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb04800dbd0 con 0x7fb058070350 2023-12-25T21:05:59.987 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.981+0000 7fb05559a700 1 -- 172.21.15.61:0/3977253343 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb048011c00 con 0x7fb058070350 2023-12-25T21:05:59.987 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.981+0000 7fb05c85a700 1 -- 172.21.15.61:0/3977253343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb058070350 msgr2=0x7fb0580fe9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:05:59.987 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.981+0000 7fb05c85a700 1 --2- 172.21.15.61:0/3977253343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb058070350 0x7fb0580fe9c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fb04800b5f0 tx=0x7fb048006ee0).stop 2023-12-25T21:05:59.987 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.981+0000 7fb05c85a700 1 -- 172.21.15.61:0/3977253343 shutdown_connections 2023-12-25T21:05:59.987 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.981+0000 7fb05c85a700 1 --2- 172.21.15.61:0/3977253343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb058070350 0x7fb0580fe9c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:05:59.987 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.981+0000 7fb05c85a700 1 -- 172.21.15.61:0/3977253343 >> 172.21.15.61:0/3977253343 conn(0x7fb0580f63c0 msgr2=0x7fb0580f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:05:59.987 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.981+0000 7fb05c85a700 1 -- 172.21.15.61:0/3977253343 shutdown_connections 2023-12-25T21:05:59.987 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.981+0000 7fb05c85a700 1 -- 172.21.15.61:0/3977253343 wait complete. 2023-12-25T21:05:59.987 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.982+0000 7fb05c85a700 1 Processor -- start 2023-12-25T21:05:59.987 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.982+0000 7fb05c85a700 1 -- start start 2023-12-25T21:05:59.987 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.982+0000 7fb05c85a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb058070350 0x7fb05810aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:05:59.988 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.982+0000 7fb05c85a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05810af40 con 0x7fb058070350 2023-12-25T21:05:59.988 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.982+0000 7fb05659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb058070350 0x7fb05810aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:05:59.988 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.982+0000 7fb05659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb058070350 0x7fb05810aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:39756/0 (socket says 172.21.15.61:39756) 2023-12-25T21:05:59.988 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.982+0000 7fb05659c700 1 -- 172.21.15.61:0/2957363003 learned_addr learned my addr 172.21.15.61:0/2957363003 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:05:59.988 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.982+0000 7fb05659c700 1 -- 172.21.15.61:0/2957363003 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04800b200 con 0x7fb058070350 2023-12-25T21:05:59.988 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.983+0000 7fb05659c700 1 --2- 172.21.15.61:0/2957363003 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb058070350 0x7fb05810aa50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb048002f40 tx=0x7fb048006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:05:59.988 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.983+0000 7fb04effd700 1 -- 172.21.15.61:0/2957363003 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb048011690 con 0x7fb058070350 2023-12-25T21:05:59.988 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.983+0000 7fb04effd700 1 -- 172.21.15.61:0/2957363003 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb04801b880 con 0x7fb058070350 2023-12-25T21:05:59.988 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.983+0000 7fb04effd700 1 -- 172.21.15.61:0/2957363003 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb04801a770 con 0x7fb058070350 2023-12-25T21:05:59.988 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.983+0000 7fb05c85a700 1 -- 172.21.15.61:0/2957363003 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb05810b140 con 0x7fb058070350 2023-12-25T21:05:59.988 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.983+0000 7fb05c85a700 1 -- 172.21.15.61:0/2957363003 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb05810b550 con 0x7fb058070350 2023-12-25T21:05:59.988 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.983+0000 7fb04effd700 1 -- 172.21.15.61:0/2957363003 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fb04800f070 con 0x7fb058070350 2023-12-25T21:05:59.988 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.983+0000 7fb04effd700 1 -- 172.21.15.61:0/2957363003 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb048025a40 con 0x7fb058070350 2023-12-25T21:05:59.988 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.984+0000 7fb05c85a700 1 -- 172.21.15.61:0/2957363003 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb058045ad0 con 0x7fb058070350 2023-12-25T21:05:59.989 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:05:59.986+0000 7fb04effd700 1 -- 172.21.15.61:0/2957363003 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fb048016070 con 0x7fb058070350 2023-12-25T21:06:00.054 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:00.054+0000 7fb05c85a700 1 -- 172.21.15.61:0/2957363003 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fb058104830 con 0x7fb058070350 2023-12-25T21:06:00.055 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:00.054+0000 7fb04effd700 1 -- 172.21.15.61:0/2957363003 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fb04801bd60 con 0x7fb058070350 2023-12-25T21:06:00.056 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:00.055+0000 7fb05c85a700 1 -- 172.21.15.61:0/2957363003 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb058070350 msgr2=0x7fb05810aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:00.056 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:00.055+0000 7fb05c85a700 1 --2- 172.21.15.61:0/2957363003 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb058070350 0x7fb05810aa50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb048002f40 tx=0x7fb048006ee0).stop 2023-12-25T21:06:00.056 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:00.055+0000 7fb05c85a700 1 -- 172.21.15.61:0/2957363003 shutdown_connections 2023-12-25T21:06:00.056 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:00.055+0000 7fb05c85a700 1 --2- 172.21.15.61:0/2957363003 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb058070350 0x7fb05810aa50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:00.056 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:00.055+0000 7fb05c85a700 1 -- 172.21.15.61:0/2957363003 >> 172.21.15.61:0/2957363003 conn(0x7fb0580f63c0 msgr2=0x7fb0580f7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:00.056 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:00.055+0000 7fb05c85a700 1 -- 172.21.15.61:0/2957363003 shutdown_connections 2023-12-25T21:06:00.056 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:00.055+0000 7fb05c85a700 1 -- 172.21.15.61:0/2957363003 wait complete. 2023-12-25T21:06:00.205 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:05:59 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111311]: audit 2023-12-25T21:05:59.173124+0000 mon.smithi061 (mon.0) 8 : audit [INF] from='client.? 172.21.15.61:0/611502147' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-25T21:06:00.205 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:05:59 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111311]: audit 2023-12-25T21:05:59.176254+0000 mon.smithi061 (mon.0) 9 : audit [INF] from='client.? 172.21.15.61:0/611502147' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-25T21:06:00.450 INFO:teuthology.orchestra.run.smithi061.stderr:Restarting the monitor... 2023-12-25T21:06:00.450 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl restart ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi061 2023-12-25T21:06:00.540 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:00 smithi061 systemd[1]: Stopping Ceph mon.smithi061 for 5532d712-a369-11ee-95a5-87774f69a715... 2023-12-25T21:06:00.823 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:00 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111311]: debug 2023-12-25T21:06:00.539+0000 7f4ae5a34700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi061 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-25T21:06:00.824 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:00 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111311]: debug 2023-12-25T21:06:00.539+0000 7f4ae5a34700 -1 mon.smithi061@0(leader) e1 *** Got Signal Terminated *** 2023-12-25T21:06:01.186 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:00 smithi061 podman[111635]: ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061 2023-12-25T21:06:01.186 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:00 smithi061 systemd[1]: ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi061.service: Succeeded. 2023-12-25T21:06:01.186 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:00 smithi061 systemd[1]: Stopped Ceph mon.smithi061 for 5532d712-a369-11ee-95a5-87774f69a715. 2023-12-25T21:06:01.186 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:00 smithi061 systemd[1]: Starting Ceph mon.smithi061 for 5532d712-a369-11ee-95a5-87774f69a715... 2023-12-25T21:06:01.251 INFO:teuthology.orchestra.run.smithi061.stderr:Setting mon public_network to 172.21.0.0/20 2023-12-25T21:06:01.251 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2023-12-25T21:06:01.609 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.608+0000 7f258503d700 1 Processor -- start 2023-12-25T21:06:01.612 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.608+0000 7f258503d700 1 -- start start 2023-12-25T21:06:01.612 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.608+0000 7f258503d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2580070390 0x7f2580100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:01.612 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.608+0000 7f258503d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25800fc770 con 0x7f2580070390 2023-12-25T21:06:01.613 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.609+0000 7f257ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2580070390 0x7f2580100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:01.613 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.609+0000 7f257ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2580070390 0x7f2580100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:39762/0 (socket says 172.21.15.61:39762) 2023-12-25T21:06:01.613 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.609+0000 7f257ed9d700 1 -- 172.21.15.61:0/2138475460 learned_addr learned my addr 172.21.15.61:0/2138475460 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:01.613 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.609+0000 7f257ed9d700 1 -- 172.21.15.61:0/2138475460 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2580101270 con 0x7f2580070390 2023-12-25T21:06:01.613 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.609+0000 7f257ed9d700 1 --2- 172.21.15.61:0/2138475460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2580070390 0x7f2580100d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f257400b5f0 tx=0x7f2574006ee0).ready entity=mon.0 client_cookie=3ffc441915c580ce server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:01.613 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.609+0000 7f257dd9b700 1 -- 172.21.15.61:0/2138475460 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f257400d600 con 0x7f2580070390 2023-12-25T21:06:01.613 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.609+0000 7f257dd9b700 1 -- 172.21.15.61:0/2138475460 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f257400dbd0 con 0x7f2580070390 2023-12-25T21:06:01.613 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.609+0000 7f257dd9b700 1 -- 172.21.15.61:0/2138475460 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2574011c00 con 0x7f2580070390 2023-12-25T21:06:01.613 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.610+0000 7f258503d700 1 -- 172.21.15.61:0/2138475460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2580070390 msgr2=0x7f2580100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:01.613 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.610+0000 7f258503d700 1 --2- 172.21.15.61:0/2138475460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2580070390 0x7f2580100d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f257400b5f0 tx=0x7f2574006ee0).stop 2023-12-25T21:06:01.613 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.610+0000 7f258503d700 1 -- 172.21.15.61:0/2138475460 shutdown_connections 2023-12-25T21:06:01.613 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.610+0000 7f258503d700 1 --2- 172.21.15.61:0/2138475460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2580070390 0x7f2580100d80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:01.614 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.610+0000 7f258503d700 1 -- 172.21.15.61:0/2138475460 >> 172.21.15.61:0/2138475460 conn(0x7f25800f63c0 msgr2=0x7f25800f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:01.614 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.610+0000 7f258503d700 1 -- 172.21.15.61:0/2138475460 shutdown_connections 2023-12-25T21:06:01.614 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.610+0000 7f258503d700 1 -- 172.21.15.61:0/2138475460 wait complete. 2023-12-25T21:06:01.614 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.611+0000 7f258503d700 1 Processor -- start 2023-12-25T21:06:01.614 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.611+0000 7f258503d700 1 -- start start 2023-12-25T21:06:01.614 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.611+0000 7f258503d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2580070390 0x7f258010cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:01.614 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.611+0000 7f258503d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258010d0f0 con 0x7f2580070390 2023-12-25T21:06:01.614 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.611+0000 7f257ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2580070390 0x7f258010cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:01.614 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.611+0000 7f257ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2580070390 0x7f258010cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:39764/0 (socket says 172.21.15.61:39764) 2023-12-25T21:06:01.614 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.611+0000 7f257ed9d700 1 -- 172.21.15.61:0/3110117354 learned_addr learned my addr 172.21.15.61:0/3110117354 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:01.615 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.612+0000 7f257ed9d700 1 -- 172.21.15.61:0/3110117354 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257400b200 con 0x7f2580070390 2023-12-25T21:06:01.615 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.612+0000 7f257ed9d700 1 --2- 172.21.15.61:0/3110117354 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2580070390 0x7f258010cc00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f257400b190 tx=0x7f2574006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:01.615 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.612+0000 7f25677fe700 1 -- 172.21.15.61:0/3110117354 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f257401b410 con 0x7f2580070390 2023-12-25T21:06:01.615 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.612+0000 7f25677fe700 1 -- 172.21.15.61:0/3110117354 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f257401b9e0 con 0x7f2580070390 2023-12-25T21:06:01.615 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.612+0000 7f258503d700 1 -- 172.21.15.61:0/3110117354 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f258010d2f0 con 0x7f2580070390 2023-12-25T21:06:01.615 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.612+0000 7f25677fe700 1 -- 172.21.15.61:0/3110117354 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f257401a880 con 0x7f2580070390 2023-12-25T21:06:01.616 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.612+0000 7f258503d700 1 -- 172.21.15.61:0/3110117354 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f258010d700 con 0x7f2580070390 2023-12-25T21:06:01.616 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.612+0000 7f25677fe700 1 -- 172.21.15.61:0/3110117354 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f257400f070 con 0x7f2580070390 2023-12-25T21:06:01.616 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.612+0000 7f25677fe700 1 -- 172.21.15.61:0/3110117354 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2574025b70 con 0x7f2580070390 2023-12-25T21:06:01.616 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.613+0000 7f258503d700 1 -- 172.21.15.61:0/3110117354 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2580045ad0 con 0x7f2580070390 2023-12-25T21:06:01.616 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.615+0000 7f25677fe700 1 -- 172.21.15.61:0/3110117354 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f2574016070 con 0x7f2580070390 2023-12-25T21:06:01.683 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.682+0000 7f258503d700 1 -- 172.21.15.61:0/3110117354 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f2580106a70 con 0x7f2580070390 2023-12-25T21:06:01.688 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.687+0000 7f25677fe700 1 -- 172.21.15.61:0/3110117354 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f2574025db0 con 0x7f2580070390 2023-12-25T21:06:01.689 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.688+0000 7f258503d700 1 -- 172.21.15.61:0/3110117354 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2580070390 msgr2=0x7f258010cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:01.689 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.688+0000 7f258503d700 1 --2- 172.21.15.61:0/3110117354 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2580070390 0x7f258010cc00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f257400b190 tx=0x7f2574006ee0).stop 2023-12-25T21:06:01.689 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.688+0000 7f258503d700 1 -- 172.21.15.61:0/3110117354 shutdown_connections 2023-12-25T21:06:01.689 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.688+0000 7f258503d700 1 --2- 172.21.15.61:0/3110117354 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2580070390 0x7f258010cc00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:01.689 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.688+0000 7f258503d700 1 -- 172.21.15.61:0/3110117354 >> 172.21.15.61:0/3110117354 conn(0x7f25800f63c0 msgr2=0x7f25800f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:01.689 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.688+0000 7f258503d700 1 -- 172.21.15.61:0/3110117354 shutdown_connections 2023-12-25T21:06:01.689 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:01.688+0000 7f258503d700 1 -- 172.21.15.61:0/3110117354 wait complete. 2023-12-25T21:06:01.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 bash[111707]: acaac92d6f689786510df427b7f3df685e3e80fe48d9688bf25b1c284097204d 2023-12-25T21:06:01.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.246+0000 7f834c558700 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-25T21:06:01.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.246+0000 7f834c558700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2023-12-25T21:06:01.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.246+0000 7f834c558700 0 pidfile_write: ignore empty --pid-file 2023-12-25T21:06:01.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 systemd[1]: Started Ceph mon.smithi061 for 5532d712-a369-11ee-95a5-87774f69a715. 2023-12-25T21:06:01.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.251+0000 7f834c558700 0 load: jerasure load: lrc load: isa 2023-12-25T21:06:01.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: RocksDB version: 6.8.1 2023-12-25T21:06:01.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-25T21:06:01.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Compile date May 13 2021 2023-12-25T21:06:01.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: DB SUMMARY 2023-12-25T21:06:01.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: CURRENT file: CURRENT 2023-12-25T21:06:01.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: IDENTITY file: IDENTITY 2023-12-25T21:06:01.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-12-25T21:06:01.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi061/store.db dir, Total Num: 1, files: 000004.sst 2023-12-25T21:06:01.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi061/store.db: 000006.log size: 92912 ; 2023-12-25T21:06:01.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.error_if_exists: 0 2023-12-25T21:06:01.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.create_if_missing: 0 2023-12-25T21:06:01.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.paranoid_checks: 1 2023-12-25T21:06:01.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.env: 0x5603ce5bc1c0 2023-12-25T21:06:01.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.fs: Posix File System 2023-12-25T21:06:01.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.info_log: 0x5603cf665960 2023-12-25T21:06:01.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-25T21:06:01.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.statistics: (nil) 2023-12-25T21:06:01.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.use_fsync: 0 2023-12-25T21:06:01.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_log_file_size: 0 2023-12-25T21:06:01.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-25T21:06:01.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-25T21:06:01.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-25T21:06:01.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-25T21:06:01.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.allow_fallocate: 1 2023-12-25T21:06:01.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-25T21:06:01.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-25T21:06:01.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.use_direct_reads: 0 2023-12-25T21:06:01.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-25T21:06:01.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.create_missing_column_families: 0 2023-12-25T21:06:01.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.db_log_dir: 2023-12-25T21:06:01.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi061/store.db 2023-12-25T21:06:01.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-25T21:06:01.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_subcompactions: 1 2023-12-25T21:06:01.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_background_flushes: -1 2023-12-25T21:06:01.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-25T21:06:01.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-25T21:06:01.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-25T21:06:01.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-25T21:06:01.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-25T21:06:01.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.advise_random_on_open: 1 2023-12-25T21:06:01.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-25T21:06:01.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.write_buffer_manager: 0x5603d02da030 2023-12-25T21:06:01.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-25T21:06:01.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-25T21:06:01.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-25T21:06:01.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-25T21:06:01.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.rate_limiter: (nil) 2023-12-25T21:06:01.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-25T21:06:01.711 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-25T21:06:01.711 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-25T21:06:01.711 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-25T21:06:01.711 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.unordered_write: 0 2023-12-25T21:06:01.713 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-25T21:06:01.713 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-25T21:06:01.713 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-25T21:06:01.713 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-25T21:06:01.713 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.row_cache: None 2023-12-25T21:06:01.713 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.wal_filter: None 2023-12-25T21:06:01.713 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-25T21:06:01.713 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-25T21:06:01.713 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.preserve_deletes: 0 2023-12-25T21:06:01.713 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.two_write_queues: 0 2023-12-25T21:06:01.713 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.manual_wal_flush: 0 2023-12-25T21:06:01.713 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.atomic_flush: 0 2023-12-25T21:06:01.714 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-25T21:06:01.714 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-25T21:06:01.714 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-25T21:06:01.714 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.log_readahead_size: 0 2023-12-25T21:06:01.714 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-12-25T21:06:01.714 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_background_jobs: 2 2023-12-25T21:06:01.714 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_background_compactions: -1 2023-12-25T21:06:01.714 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-25T21:06:01.714 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-25T21:06:01.714 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-25T21:06:01.714 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_total_wal_size: 0 2023-12-25T21:06:01.714 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-25T21:06:01.714 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-25T21:06:01.714 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-25T21:06:01.715 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-25T21:06:01.715 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_open_files: -1 2023-12-25T21:06:01.715 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.bytes_per_sync: 0 2023-12-25T21:06:01.715 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-25T21:06:01.715 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-25T21:06:01.715 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-25T21:06:01.715 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Compression algorithms supported: 2023-12-25T21:06:01.715 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-25T21:06:01.715 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: kZSTD supported: 0 2023-12-25T21:06:01.715 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: kXpressCompression supported: 0 2023-12-25T21:06:01.715 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-25T21:06:01.715 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: kLZ4Compression supported: 1 2023-12-25T21:06:01.715 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: kBZip2Compression supported: 0 2023-12-25T21:06:01.715 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: kZlibCompression supported: 1 2023-12-25T21:06:01.716 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: kSnappyCompression supported: 1 2023-12-25T21:06:01.716 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-25T21:06:01.716 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi061/store.db/MANIFEST-000005 2023-12-25T21:06:01.716 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.716 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-12-25T21:06:01.716 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.716 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-25T21:06:01.716 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.merge_operator: 2023-12-25T21:06:01.716 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compaction_filter: None 2023-12-25T21:06:01.716 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compaction_filter_factory: None 2023-12-25T21:06:01.716 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-25T21:06:01.716 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-25T21:06:01.716 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5603cf57c128) 2023-12-25T21:06:01.716 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cache_index_and_filter_blocks: 1 2023-12-25T21:06:01.717 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-25T21:06:01.717 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-25T21:06:01.717 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: pin_top_level_index_and_filter: 1 2023-12-25T21:06:01.717 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: index_type: 0 2023-12-25T21:06:01.717 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: data_block_index_type: 0 2023-12-25T21:06:01.717 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: index_shortening: 1 2023-12-25T21:06:01.717 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: data_block_hash_table_util_ratio: 0.750000 2023-12-25T21:06:01.717 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: hash_index_allow_collision: 1 2023-12-25T21:06:01.717 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: checksum: 1 2023-12-25T21:06:01.717 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: no_block_cache: 0 2023-12-25T21:06:01.717 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: block_cache: 0x5603cf5b2f10 2023-12-25T21:06:01.717 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: block_cache_name: BinnedLRUCache 2023-12-25T21:06:01.717 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: block_cache_options: 2023-12-25T21:06:01.718 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: capacity : 536870912 2023-12-25T21:06:01.718 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: num_shard_bits : 4 2023-12-25T21:06:01.718 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: strict_capacity_limit : 0 2023-12-25T21:06:01.718 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: high_pri_pool_ratio: 0.000 2023-12-25T21:06:01.718 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: block_cache_compressed: (nil) 2023-12-25T21:06:01.718 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: persistent_cache: (nil) 2023-12-25T21:06:01.718 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: block_size: 4096 2023-12-25T21:06:01.718 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: block_size_deviation: 10 2023-12-25T21:06:01.718 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: block_restart_interval: 16 2023-12-25T21:06:01.718 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: index_block_restart_interval: 1 2023-12-25T21:06:01.718 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: metadata_block_size: 4096 2023-12-25T21:06:01.718 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: partition_filters: 0 2023-12-25T21:06:01.718 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: use_delta_encoding: 1 2023-12-25T21:06:01.718 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-25T21:06:01.719 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: whole_key_filtering: 1 2023-12-25T21:06:01.719 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: verify_compression: 0 2023-12-25T21:06:01.719 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: read_amp_bytes_per_bit: 0 2023-12-25T21:06:01.719 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: format_version: 2 2023-12-25T21:06:01.719 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: enable_index_compression: 1 2023-12-25T21:06:01.719 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: block_align: 0 2023-12-25T21:06:01.719 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.719 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-25T21:06:01.719 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-25T21:06:01.719 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compression: NoCompression 2023-12-25T21:06:01.719 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-25T21:06:01.719 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-25T21:06:01.719 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-25T21:06:01.719 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.num_levels: 7 2023-12-25T21:06:01.720 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-25T21:06:01.720 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-25T21:06:01.720 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-25T21:06:01.720 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-25T21:06:01.720 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-25T21:06:01.720 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-25T21:06:01.720 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-25T21:06:01.720 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-25T21:06:01.720 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-25T21:06:01.720 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-25T21:06:01.720 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compression_opts.level: 32767 2023-12-25T21:06:01.720 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-25T21:06:01.720 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-25T21:06:01.720 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-25T21:06:01.721 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compression_opts.enabled: false 2023-12-25T21:06:01.721 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-25T21:06:01.721 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-25T21:06:01.721 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-25T21:06:01.721 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-25T21:06:01.721 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-25T21:06:01.721 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-25T21:06:01.721 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-25T21:06:01.721 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-25T21:06:01.721 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-25T21:06:01.721 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-25T21:06:01.721 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-25T21:06:01.721 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-25T21:06:01.722 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-25T21:06:01.722 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-25T21:06:01.722 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-25T21:06:01.722 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-25T21:06:01.722 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-25T21:06:01.722 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.arena_block_size: 4194304 2023-12-25T21:06:01.722 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-25T21:06:01.722 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-25T21:06:01.722 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-25T21:06:01.722 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-25T21:06:01.722 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-25T21:06:01.722 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-25T21:06:01.722 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-25T21:06:01.723 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-25T21:06:01.723 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-25T21:06:01.723 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-25T21:06:01.723 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-25T21:06:01.723 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-25T21:06:01.723 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-25T21:06:01.723 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-25T21:06:01.723 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.table_properties_collectors: 2023-12-25T21:06:01.723 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.inplace_update_support: 0 2023-12-25T21:06:01.723 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-25T21:06:01.723 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-25T21:06:01.723 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-25T21:06:01.723 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-25T21:06:01.723 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.bloom_locality: 0 2023-12-25T21:06:01.724 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.max_successive_merges: 0 2023-12-25T21:06:01.724 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-25T21:06:01.724 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-25T21:06:01.724 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.force_consistency_checks: 0 2023-12-25T21:06:01.724 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-25T21:06:01.724 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.ttl: 2592000 2023-12-25T21:06:01.724 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.252+0000 7f834c558700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-25T21:06:01.724 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.253+0000 7f834c558700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi061/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-25T21:06:01.724 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.724 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.253+0000 7f834c558700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-12-25T21:06:01.724 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.724 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.253+0000 7f834c558700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1703538361254519, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-12-25T21:06:01.724 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.253+0000 7f834c558700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-12-25T21:06:01.725 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.254+0000 7f834c558700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-12-25T21:06:01.725 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.254+0000 7f834c558700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1703538361255909, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88692, "table_properties": {"data_size": 86807, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79948, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1703538361, "oldest_key_time": 3, "file_creation_time": 0}} 2023-12-25T21:06:01.725 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.255+0000 7f834c558700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-12-25T21:06:01.725 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.725 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.256+0000 7f834c558700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1703538361257156, "job": 1, "event": "recovery_finished"} 2023-12-25T21:06:01.725 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.257+0000 7f834c558700 4 rocksdb: DB pointer 0x5603cf677800 2023-12-25T21:06:01.725 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.257+0000 7f8333370700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-12-25T21:06:01.725 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.257+0000 7f8333370700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-12-25T21:06:01.725 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: ** DB Stats ** 2023-12-25T21:06:01.725 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-25T21:06:01.725 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-25T21:06:01.725 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-25T21:06:01.725 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-25T21:06:01.726 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-25T21:06:01.726 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-25T21:06:01.726 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-25T21:06:01.726 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.726 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: ** Compaction Stats [default] ** 2023-12-25T21:06:01.726 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-25T21:06:01.726 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-25T21:06:01.726 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 95.3 0.00 0.00 1 0.001 0 0 2023-12-25T21:06:01.726 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 95.3 0.00 0.00 1 0.001 0 0 2023-12-25T21:06:01.726 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 95.3 0.00 0.00 1 0.001 0 0 2023-12-25T21:06:01.726 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.726 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: ** Compaction Stats [default] ** 2023-12-25T21:06:01.726 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-25T21:06:01.727 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-25T21:06:01.727 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 95.3 0.00 0.00 1 0.001 0 0 2023-12-25T21:06:01.727 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-25T21:06:01.727 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-25T21:06:01.727 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-25T21:06:01.727 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: AddFile(Total Files): cumulative 0, interval 0 2023-12-25T21:06:01.727 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-25T21:06:01.727 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: AddFile(Keys): cumulative 0, interval 0 2023-12-25T21:06:01.727 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Cumulative compaction: 0.00 GB write, 17.93 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-25T21:06:01.727 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Interval compaction: 0.00 GB write, 17.93 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-25T21:06:01.727 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-25T21:06:01.727 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.727 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: ** File Read Latency Histogram By Level [default] ** 2023-12-25T21:06:01.727 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.728 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: ** Compaction Stats [default] ** 2023-12-25T21:06:01.728 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-25T21:06:01.728 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-25T21:06:01.728 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 95.3 0.00 0.00 1 0.001 0 0 2023-12-25T21:06:01.728 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 95.3 0.00 0.00 1 0.001 0 0 2023-12-25T21:06:01.728 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-25T21:06:01.728 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.728 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: ** Compaction Stats [default] ** 2023-12-25T21:06:01.728 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-25T21:06:01.728 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-25T21:06:01.728 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 95.3 0.00 0.00 1 0.001 0 0 2023-12-25T21:06:01.728 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-25T21:06:01.728 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-25T21:06:01.729 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-25T21:06:01.729 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: AddFile(Total Files): cumulative 0, interval 0 2023-12-25T21:06:01.729 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-25T21:06:01.729 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: AddFile(Keys): cumulative 0, interval 0 2023-12-25T21:06:01.729 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Cumulative compaction: 0.00 GB write, 17.75 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-25T21:06:01.729 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-25T21:06:01.729 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-25T21:06:01.729 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.729 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: ** File Read Latency Histogram By Level [default] ** 2023-12-25T21:06:01.729 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.729 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.258+0000 7f834c558700 0 starting mon.smithi061 rank 0 at public addrs [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] at bind addrs [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi061 fsid 5532d712-a369-11ee-95a5-87774f69a715 2023-12-25T21:06:01.729 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.259+0000 7f834c558700 1 mon.smithi061@-1(???) e1 preinit fsid 5532d712-a369-11ee-95a5-87774f69a715 2023-12-25T21:06:01.729 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.260+0000 7f834c558700 0 mon.smithi061@-1(???).mds e1 new map 2023-12-25T21:06:01.730 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.260+0000 7f834c558700 0 mon.smithi061@-1(???).mds e1 print_map 2023-12-25T21:06:01.730 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: e1 2023-12-25T21:06:01.730 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: enable_multiple, ever_enabled_multiple: 1,1 2023-12-25T21:06:01.730 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-25T21:06:01.730 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: legacy client fscid: -1 2023-12-25T21:06:01.730 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.730 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: No filesystems configured 2023-12-25T21:06:01.730 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: 2023-12-25T21:06:01.730 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.260+0000 7f834c558700 0 mon.smithi061@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-12-25T21:06:01.730 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.260+0000 7f834c558700 0 mon.smithi061@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-25T21:06:01.730 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.260+0000 7f834c558700 0 mon.smithi061@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-25T21:06:01.730 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.260+0000 7f834c558700 0 mon.smithi061@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-25T21:06:01.730 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:05:59.173124+0000 mon.smithi061 (mon.0) 8 : audit [INF] from='client.? 172.21.15.61:0/611502147' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-25T21:06:01.730 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:05:59.176254+0000 mon.smithi061 (mon.0) 9 : audit [INF] from='client.? 172.21.15.61:0/611502147' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-25T21:06:01.731 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.260+0000 7f834c558700 1 mon.smithi061@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-12-25T21:06:01.731 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.262+0000 7f834c558700 20 expand_channel_meta expand map: {default=false} 2023-12-25T21:06:01.731 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.262+0000 7f834c558700 20 expand_channel_meta from 'false' to 'false' 2023-12-25T21:06:01.731 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.262+0000 7f834c558700 20 expand_channel_meta expanded map: {default=false} 2023-12-25T21:06:01.731 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.262+0000 7f834c558700 20 expand_channel_meta expand map: {default=info} 2023-12-25T21:06:01.731 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.262+0000 7f834c558700 20 expand_channel_meta from 'info' to 'info' 2023-12-25T21:06:01.731 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.262+0000 7f834c558700 20 expand_channel_meta expanded map: {default=info} 2023-12-25T21:06:01.731 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.262+0000 7f834c558700 20 expand_channel_meta expand map: {default=daemon} 2023-12-25T21:06:01.731 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.262+0000 7f834c558700 20 expand_channel_meta from 'daemon' to 'daemon' 2023-12-25T21:06:01.731 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.262+0000 7f834c558700 20 expand_channel_meta expanded map: {default=daemon} 2023-12-25T21:06:01.731 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.262+0000 7f834c558700 20 expand_channel_meta expand map: {default=debug} 2023-12-25T21:06:01.731 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.262+0000 7f834c558700 20 expand_channel_meta from 'debug' to 'debug' 2023-12-25T21:06:01.731 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:06:01.262+0000 7f834c558700 20 expand_channel_meta expanded map: {default=debug} 2023-12-25T21:06:01.731 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:01.277765+0000 mon.smithi061 (mon.0) 1 : cluster [INF] mon.smithi061 is new leader, mons smithi061 in quorum (ranks 0) 2023-12-25T21:06:01.731 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:01.277877+0000 mon.smithi061 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi061=[v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0]} 2023-12-25T21:06:01.732 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:01.277945+0000 mon.smithi061 (mon.0) 3 : cluster [DBG] fsmap 2023-12-25T21:06:01.732 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:01.277992+0000 mon.smithi061 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-12-25T21:06:01.732 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:01.278642+0000 mon.smithi061 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-12-25T21:06:02.102 INFO:teuthology.orchestra.run.smithi061.stderr:Wrote config to /etc/ceph/ceph.conf 2023-12-25T21:06:02.102 INFO:teuthology.orchestra.run.smithi061.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-12-25T21:06:02.102 INFO:teuthology.orchestra.run.smithi061.stderr:Creating mgr... 2023-12-25T21:06:02.103 INFO:teuthology.orchestra.run.smithi061.stderr:Verifying port 9283 ... 2023-12-25T21:06:02.104 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl daemon-reload 2023-12-25T21:06:02.206 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl stop ceph-5532d712-a369-11ee-95a5-87774f69a715@mgr.smithi061.jxxeke 2023-12-25T21:06:02.240 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl reset-failed ceph-5532d712-a369-11ee-95a5-87774f69a715@mgr.smithi061.jxxeke 2023-12-25T21:06:02.246 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Failed to reset failed state of unit ceph-5532d712-a369-11ee-95a5-87774f69a715@mgr.smithi061.jxxeke.service: Unit ceph-5532d712-a369-11ee-95a5-87774f69a715@mgr.smithi061.jxxeke.service not loaded. 2023-12-25T21:06:02.246 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl enable ceph-5532d712-a369-11ee-95a5-87774f69a715@mgr.smithi061.jxxeke 2023-12-25T21:06:02.252 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: Created symlink /etc/systemd/system/ceph-5532d712-a369-11ee-95a5-87774f69a715.target.wants/ceph-5532d712-a369-11ee-95a5-87774f69a715@mgr.smithi061.jxxeke.service → /etc/systemd/system/ceph-5532d712-a369-11ee-95a5-87774f69a715@.service. 2023-12-25T21:06:02.344 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl start ceph-5532d712-a369-11ee-95a5-87774f69a715@mgr.smithi061.jxxeke 2023-12-25T21:06:02.710 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-25T21:06:02.719 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: disabled 2023-12-25T21:06:02.720 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl is-active firewalld.service 2023-12-25T21:06:02.728 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: inactive 2023-12-25T21:06:02.728 INFO:teuthology.orchestra.run.smithi061.stderr:firewalld.service is not enabled 2023-12-25T21:06:02.728 INFO:teuthology.orchestra.run.smithi061.stderr:Not possible to enable service . firewalld.service is not available 2023-12-25T21:06:02.729 INFO:teuthology.orchestra.run.smithi061.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-12-25T21:06:02.729 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-25T21:06:02.738 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: disabled 2023-12-25T21:06:02.738 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl is-active firewalld.service 2023-12-25T21:06:02.747 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: inactive 2023-12-25T21:06:02.748 INFO:teuthology.orchestra.run.smithi061.stderr:firewalld.service is not enabled 2023-12-25T21:06:02.748 INFO:teuthology.orchestra.run.smithi061.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-12-25T21:06:02.748 INFO:teuthology.orchestra.run.smithi061.stderr:Waiting for mgr to start... 2023-12-25T21:06:02.748 INFO:teuthology.orchestra.run.smithi061.stderr:Waiting for mgr... 2023-12-25T21:06:02.748 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-25T21:06:02.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:02 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:01.688688+0000 mon.smithi061 (mon.0) 6 : audit [INF] from='client.? 172.21.15.61:0/3110117354' entity='client.admin' 2023-12-25T21:06:03.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.106+0000 7f2c198a6700 1 Processor -- start 2023-12-25T21:06:03.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.106+0000 7f2c198a6700 1 -- start start 2023-12-25T21:06:03.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.106+0000 7f2c198a6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c140703b0 0x7f2c14100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:03.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.106+0000 7f2c198a6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c140fc5a0 con 0x7f2c140703b0 2023-12-25T21:06:03.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.106+0000 7f2c188a4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c140703b0 0x7f2c14100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:03.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.106+0000 7f2c188a4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c140703b0 0x7f2c14100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:39800/0 (socket says 172.21.15.61:39800) 2023-12-25T21:06:03.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.106+0000 7f2c188a4700 1 -- 172.21.15.61:0/2592248847 learned_addr learned my addr 172.21.15.61:0/2592248847 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:03.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.107+0000 7f2c188a4700 1 -- 172.21.15.61:0/2592248847 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c141010a0 con 0x7f2c140703b0 2023-12-25T21:06:03.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.107+0000 7f2c188a4700 1 --2- 172.21.15.61:0/2592248847 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c140703b0 0x7f2c14100bb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f2c0800b5f0 tx=0x7f2c08006ee0).ready entity=mon.0 client_cookie=6c4fcc022477b5ec server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:03.110 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.107+0000 7f2c137fe700 1 -- 172.21.15.61:0/2592248847 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c0800d600 con 0x7f2c140703b0 2023-12-25T21:06:03.111 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.107+0000 7f2c137fe700 1 -- 172.21.15.61:0/2592248847 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2c0800dbd0 con 0x7f2c140703b0 2023-12-25T21:06:03.111 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.107+0000 7f2c137fe700 1 -- 172.21.15.61:0/2592248847 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c08011c00 con 0x7f2c140703b0 2023-12-25T21:06:03.111 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.108+0000 7f2c198a6700 1 -- 172.21.15.61:0/2592248847 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c140703b0 msgr2=0x7f2c14100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:03.111 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.108+0000 7f2c198a6700 1 --2- 172.21.15.61:0/2592248847 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c140703b0 0x7f2c14100bb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f2c0800b5f0 tx=0x7f2c08006ee0).stop 2023-12-25T21:06:03.111 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.108+0000 7f2c198a6700 1 -- 172.21.15.61:0/2592248847 shutdown_connections 2023-12-25T21:06:03.111 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.108+0000 7f2c198a6700 1 --2- 172.21.15.61:0/2592248847 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c140703b0 0x7f2c14100bb0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:03.111 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.108+0000 7f2c198a6700 1 -- 172.21.15.61:0/2592248847 >> 172.21.15.61:0/2592248847 conn(0x7f2c140f6210 msgr2=0x7f2c140f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:03.111 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.108+0000 7f2c198a6700 1 -- 172.21.15.61:0/2592248847 shutdown_connections 2023-12-25T21:06:03.111 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.108+0000 7f2c198a6700 1 -- 172.21.15.61:0/2592248847 wait complete. 2023-12-25T21:06:03.111 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.109+0000 7f2c198a6700 1 Processor -- start 2023-12-25T21:06:03.111 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.109+0000 7f2c198a6700 1 -- start start 2023-12-25T21:06:03.111 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.109+0000 7f2c198a6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c140703b0 0x7f2c1410ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:03.112 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.109+0000 7f2c198a6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1410cf60 con 0x7f2c140703b0 2023-12-25T21:06:03.112 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.109+0000 7f2c188a4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c140703b0 0x7f2c1410ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:03.112 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.109+0000 7f2c188a4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c140703b0 0x7f2c1410ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:39804/0 (socket says 172.21.15.61:39804) 2023-12-25T21:06:03.113 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.109+0000 7f2c188a4700 1 -- 172.21.15.61:0/1580392809 learned_addr learned my addr 172.21.15.61:0/1580392809 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:03.113 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.109+0000 7f2c188a4700 1 -- 172.21.15.61:0/1580392809 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0800b200 con 0x7f2c140703b0 2023-12-25T21:06:03.113 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.110+0000 7f2c188a4700 1 --2- 172.21.15.61:0/1580392809 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c140703b0 0x7f2c1410ca70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f2c0800e040 tx=0x7f2c08006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:03.113 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.110+0000 7f2c117fa700 1 -- 172.21.15.61:0/1580392809 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c0800d9f0 con 0x7f2c140703b0 2023-12-25T21:06:03.113 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.110+0000 7f2c198a6700 1 -- 172.21.15.61:0/1580392809 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1410d160 con 0x7f2c140703b0 2023-12-25T21:06:03.113 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.110+0000 7f2c117fa700 1 -- 172.21.15.61:0/1580392809 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2c0801a9f0 con 0x7f2c140703b0 2023-12-25T21:06:03.113 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.110+0000 7f2c117fa700 1 -- 172.21.15.61:0/1580392809 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c0801b8c0 con 0x7f2c140703b0 2023-12-25T21:06:03.113 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.110+0000 7f2c198a6700 1 -- 172.21.15.61:0/1580392809 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c1410d570 con 0x7f2c140703b0 2023-12-25T21:06:03.114 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.110+0000 7f2c117fa700 1 -- 172.21.15.61:0/1580392809 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f2c0800f070 con 0x7f2c140703b0 2023-12-25T21:06:03.114 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.110+0000 7f2c117fa700 1 -- 172.21.15.61:0/1580392809 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2c08025b30 con 0x7f2c140703b0 2023-12-25T21:06:03.114 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.111+0000 7f2c198a6700 1 -- 172.21.15.61:0/1580392809 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c14045ad0 con 0x7f2c140703b0 2023-12-25T21:06:03.114 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.113+0000 7f2c117fa700 1 -- 172.21.15.61:0/1580392809 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f2c08016070 con 0x7f2c140703b0 2023-12-25T21:06:03.191 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.184+0000 7f2c198a6700 1 -- 172.21.15.61:0/1580392809 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2c141068f0 con 0x7f2c140703b0 2023-12-25T21:06:03.191 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.185+0000 7f2c117fa700 1 -- 172.21.15.61:0/1580392809 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f2c0801bd70 con 0x7f2c140703b0 2023-12-25T21:06:03.191 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.191 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: { 2023-12-25T21:06:03.191 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "fsid": "5532d712-a369-11ee-95a5-87774f69a715", 2023-12-25T21:06:03.191 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "health": { 2023-12-25T21:06:03.191 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-25T21:06:03.191 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "checks": {}, 2023-12-25T21:06:03.191 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "mutes": [] 2023-12-25T21:06:03.191 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-25T21:06:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum": [ 2023-12-25T21:06:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 0 2023-12-25T21:06:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-12-25T21:06:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-25T21:06:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "smithi061" 2023-12-25T21:06:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-12-25T21:06:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum_age": 1, 2023-12-25T21:06:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "monmap": { 2023-12-25T21:06:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T21:06:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-25T21:06:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-25T21:06:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osdmap": { 2023-12-25T21:06:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T21:06:03.193 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-25T21:06:03.193 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-25T21:06:03.193 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-25T21:06:03.193 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-25T21:06:03.193 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-25T21:06:03.193 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-25T21:06:03.193 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:03.193 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "pgmap": { 2023-12-25T21:06:03.193 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-25T21:06:03.193 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-25T21:06:03.194 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-25T21:06:03.194 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-25T21:06:03.194 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-25T21:06:03.194 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-25T21:06:03.194 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-25T21:06:03.194 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-25T21:06:03.194 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:03.194 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "fsmap": { 2023-12-25T21:06:03.194 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T21:06:03.194 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-25T21:06:03.194 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-25T21:06:03.194 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:03.194 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-25T21:06:03.194 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "available": false, 2023-12-25T21:06:03.194 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-25T21:06:03.195 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "modules": [ 2023-12-25T21:06:03.195 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "iostat", 2023-12-25T21:06:03.195 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "restful" 2023-12-25T21:06:03.195 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-12-25T21:06:03.195 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "services": {} 2023-12-25T21:06:03.195 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:03.195 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "servicemap": { 2023-12-25T21:06:03.195 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T21:06:03.195 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "modified": "2023-12-25T21:05:57.845222+0000", 2023-12-25T21:06:03.195 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "services": {} 2023-12-25T21:06:03.195 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:03.195 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-25T21:06:03.195 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: } 2023-12-25T21:06:03.195 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.186+0000 7f2c198a6700 1 -- 172.21.15.61:0/1580392809 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c140703b0 msgr2=0x7f2c1410ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:03.196 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.186+0000 7f2c198a6700 1 --2- 172.21.15.61:0/1580392809 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c140703b0 0x7f2c1410ca70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f2c0800e040 tx=0x7f2c08006ee0).stop 2023-12-25T21:06:03.196 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.186+0000 7f2c198a6700 1 -- 172.21.15.61:0/1580392809 shutdown_connections 2023-12-25T21:06:03.196 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.186+0000 7f2c198a6700 1 --2- 172.21.15.61:0/1580392809 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c140703b0 0x7f2c1410ca70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:03.196 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.186+0000 7f2c198a6700 1 -- 172.21.15.61:0/1580392809 >> 172.21.15.61:0/1580392809 conn(0x7f2c140f6210 msgr2=0x7f2c140f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:03.196 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.186+0000 7f2c198a6700 1 -- 172.21.15.61:0/1580392809 shutdown_connections 2023-12-25T21:06:03.196 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:03.186+0000 7f2c198a6700 1 -- 172.21.15.61:0/1580392809 wait complete. 2023-12-25T21:06:03.547 INFO:teuthology.orchestra.run.smithi061.stderr:mgr not available, waiting (1/15)... 2023-12-25T21:06:03.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:03.186002+0000 mon.smithi061 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.61:0/1580392809' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-25T21:06:05.547 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-25T21:06:05.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.931+0000 7fb1e13ba700 1 Processor -- start 2023-12-25T21:06:05.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.931+0000 7fb1e13ba700 1 -- start start 2023-12-25T21:06:05.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.932+0000 7fb1e13ba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:05.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.932+0000 7fb1e13ba700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc0fee10 con 0x7fb1dc0703f0 2023-12-25T21:06:05.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.932+0000 7fb1daffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:05.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.932+0000 7fb1daffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:39808/0 (socket says 172.21.15.61:39808) 2023-12-25T21:06:05.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.932+0000 7fb1daffd700 1 -- 172.21.15.61:0/4005225593 learned_addr learned my addr 172.21.15.61:0/4005225593 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:05.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.932+0000 7fb1daffd700 1 -- 172.21.15.61:0/4005225593 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1dc0fef50 con 0x7fb1dc0703f0 2023-12-25T21:06:05.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.932+0000 7fb1daffd700 1 --2- 172.21.15.61:0/4005225593 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc0fe920 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb1cc00b5f0 tx=0x7fb1cc006ee0).ready entity=mon.0 client_cookie=fa5071b685751526 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:05.934 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.932+0000 7fb1d9ffb700 1 -- 172.21.15.61:0/4005225593 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1cc00d600 con 0x7fb1dc0703f0 2023-12-25T21:06:05.937 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.932+0000 7fb1d9ffb700 1 -- 172.21.15.61:0/4005225593 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb1cc00dbd0 con 0x7fb1dc0703f0 2023-12-25T21:06:05.937 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.932+0000 7fb1d9ffb700 1 -- 172.21.15.61:0/4005225593 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1cc011c00 con 0x7fb1dc0703f0 2023-12-25T21:06:05.937 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.933+0000 7fb1e13ba700 1 -- 172.21.15.61:0/4005225593 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1dc0703f0 msgr2=0x7fb1dc0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:05.937 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.933+0000 7fb1e13ba700 1 --2- 172.21.15.61:0/4005225593 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc0fe920 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb1cc00b5f0 tx=0x7fb1cc006ee0).stop 2023-12-25T21:06:05.937 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.933+0000 7fb1e13ba700 1 -- 172.21.15.61:0/4005225593 shutdown_connections 2023-12-25T21:06:05.937 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.933+0000 7fb1e13ba700 1 --2- 172.21.15.61:0/4005225593 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc0fe920 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:05.937 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.933+0000 7fb1e13ba700 1 -- 172.21.15.61:0/4005225593 >> 172.21.15.61:0/4005225593 conn(0x7fb1dc0f63c0 msgr2=0x7fb1dc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:05.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.933+0000 7fb1e13ba700 1 -- 172.21.15.61:0/4005225593 shutdown_connections 2023-12-25T21:06:05.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.933+0000 7fb1e13ba700 1 -- 172.21.15.61:0/4005225593 wait complete. 2023-12-25T21:06:05.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.934+0000 7fb1e13ba700 1 Processor -- start 2023-12-25T21:06:05.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.934+0000 7fb1e13ba700 1 -- start start 2023-12-25T21:06:05.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.934+0000 7fb1e13ba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc0fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:05.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.934+0000 7fb1e13ba700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc0fa820 con 0x7fb1dc0703f0 2023-12-25T21:06:05.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.934+0000 7fb1daffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc0fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:05.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.934+0000 7fb1daffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc0fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:39824/0 (socket says 172.21.15.61:39824) 2023-12-25T21:06:05.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.934+0000 7fb1daffd700 1 -- 172.21.15.61:0/2997333243 learned_addr learned my addr 172.21.15.61:0/2997333243 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:05.938 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.935+0000 7fb1daffd700 1 -- 172.21.15.61:0/2997333243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1cc00b200 con 0x7fb1dc0703f0 2023-12-25T21:06:05.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.935+0000 7fb1daffd700 1 --2- 172.21.15.61:0/2997333243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc0fc3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fb1cc00b190 tx=0x7fb1cc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:05.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.935+0000 7fb1c37fe700 1 -- 172.21.15.61:0/2997333243 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1cc01b410 con 0x7fb1dc0703f0 2023-12-25T21:06:05.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.935+0000 7fb1c37fe700 1 -- 172.21.15.61:0/2997333243 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb1cc01b9e0 con 0x7fb1dc0703f0 2023-12-25T21:06:05.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.935+0000 7fb1e13ba700 1 -- 172.21.15.61:0/2997333243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1dc0fa9c0 con 0x7fb1dc0703f0 2023-12-25T21:06:05.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.935+0000 7fb1c37fe700 1 -- 172.21.15.61:0/2997333243 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1cc01a880 con 0x7fb1dc0703f0 2023-12-25T21:06:05.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.935+0000 7fb1e13ba700 1 -- 172.21.15.61:0/2997333243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1dc0fadd0 con 0x7fb1dc0703f0 2023-12-25T21:06:05.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.936+0000 7fb1c17fa700 1 -- 172.21.15.61:0/2997333243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1dc045ad0 con 0x7fb1dc0703f0 2023-12-25T21:06:05.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.936+0000 7fb1c37fe700 1 -- 172.21.15.61:0/2997333243 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fb1cc00f070 con 0x7fb1dc0703f0 2023-12-25T21:06:05.939 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.936+0000 7fb1c37fe700 1 -- 172.21.15.61:0/2997333243 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb1cc01ad50 con 0x7fb1dc0703f0 2023-12-25T21:06:05.940 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:05.938+0000 7fb1c37fe700 1 -- 172.21.15.61:0/2997333243 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fb1cc016070 con 0x7fb1dc0703f0 2023-12-25T21:06:06.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:06.007+0000 7fb1c17fa700 1 -- 172.21.15.61:0/2997333243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb1dc0ffbd0 con 0x7fb1dc0703f0 2023-12-25T21:06:06.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:06.008+0000 7fb1c37fe700 1 -- 172.21.15.61:0/2997333243 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fb1cc01b7d0 con 0x7fb1dc0703f0 2023-12-25T21:06:06.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:06.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: { 2023-12-25T21:06:06.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "fsid": "5532d712-a369-11ee-95a5-87774f69a715", 2023-12-25T21:06:06.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "health": { 2023-12-25T21:06:06.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-25T21:06:06.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "checks": {}, 2023-12-25T21:06:06.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "mutes": [] 2023-12-25T21:06:06.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:06.015 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-25T21:06:06.015 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum": [ 2023-12-25T21:06:06.015 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 0 2023-12-25T21:06:06.015 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-12-25T21:06:06.015 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-25T21:06:06.015 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "smithi061" 2023-12-25T21:06:06.015 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-12-25T21:06:06.015 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum_age": 4, 2023-12-25T21:06:06.015 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "monmap": { 2023-12-25T21:06:06.015 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T21:06:06.015 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-25T21:06:06.015 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-25T21:06:06.015 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:06.015 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osdmap": { 2023-12-25T21:06:06.016 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T21:06:06.016 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-25T21:06:06.016 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-25T21:06:06.016 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-25T21:06:06.016 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-25T21:06:06.016 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-25T21:06:06.016 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-25T21:06:06.016 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:06.016 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "pgmap": { 2023-12-25T21:06:06.016 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-25T21:06:06.017 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-25T21:06:06.017 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-25T21:06:06.017 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-25T21:06:06.017 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-25T21:06:06.017 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-25T21:06:06.017 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-25T21:06:06.017 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-25T21:06:06.017 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:06.018 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "fsmap": { 2023-12-25T21:06:06.018 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T21:06:06.018 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-25T21:06:06.018 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-25T21:06:06.018 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:06.018 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-25T21:06:06.018 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "available": false, 2023-12-25T21:06:06.018 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-25T21:06:06.018 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "modules": [ 2023-12-25T21:06:06.018 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "iostat", 2023-12-25T21:06:06.019 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "restful" 2023-12-25T21:06:06.019 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-12-25T21:06:06.019 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "services": {} 2023-12-25T21:06:06.019 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:06.019 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "servicemap": { 2023-12-25T21:06:06.019 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T21:06:06.019 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "modified": "2023-12-25T21:05:57.845222+0000", 2023-12-25T21:06:06.019 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "services": {} 2023-12-25T21:06:06.019 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:06.019 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-25T21:06:06.020 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: } 2023-12-25T21:06:06.020 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:06.008+0000 7fb1c17fa700 1 -- 172.21.15.61:0/2997333243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1dc0703f0 msgr2=0x7fb1dc0fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:06.020 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:06.008+0000 7fb1c17fa700 1 --2- 172.21.15.61:0/2997333243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc0fc3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fb1cc00b190 tx=0x7fb1cc006ee0).stop 2023-12-25T21:06:06.020 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:06.008+0000 7fb1c17fa700 1 -- 172.21.15.61:0/2997333243 shutdown_connections 2023-12-25T21:06:06.020 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:06.008+0000 7fb1c17fa700 1 --2- 172.21.15.61:0/2997333243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc0fc3a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:06.020 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:06.008+0000 7fb1c17fa700 1 -- 172.21.15.61:0/2997333243 >> 172.21.15.61:0/2997333243 conn(0x7fb1dc0f63c0 msgr2=0x7fb1dc0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:06.020 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:06.009+0000 7fb1c17fa700 1 -- 172.21.15.61:0/2997333243 shutdown_connections 2023-12-25T21:06:06.020 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:06.009+0000 7fb1c17fa700 1 -- 172.21.15.61:0/2997333243 wait complete. 2023-12-25T21:06:06.360 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:06 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:06.008798+0000 mon.smithi061 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.61:0/2997333243' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-25T21:06:06.382 INFO:teuthology.orchestra.run.smithi061.stderr:mgr not available, waiting (2/15)... 2023-12-25T21:06:08.205 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:07 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:07.889760+0000 mon.smithi061 (mon.0) 9 : cluster [INF] Activating manager daemon smithi061.jxxeke 2023-12-25T21:06:08.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:07 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:07.892349+0000 mon.smithi061 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi061.jxxeke(active, starting, since 0.00271012s) 2023-12-25T21:06:08.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:07 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:07.893008+0000 mon.smithi061 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.61:0/2054091183' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-25T21:06:08.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:07 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:07.893159+0000 mon.smithi061 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.61:0/2054091183' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-25T21:06:08.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:07 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:07.893344+0000 mon.smithi061 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.61:0/2054091183' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-25T21:06:08.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:07 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:07.893541+0000 mon.smithi061 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.61:0/2054091183' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "mon metadata", "id": "smithi061"}]: dispatch 2023-12-25T21:06:08.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:07 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:07.893821+0000 mon.smithi061 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.61:0/2054091183' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "mgr metadata", "who": "smithi061.jxxeke", "id": "smithi061.jxxeke"}]: dispatch 2023-12-25T21:06:08.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:07 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:07.907349+0000 mon.smithi061 (mon.0) 16 : cluster [INF] Manager daemon smithi061.jxxeke is now available 2023-12-25T21:06:08.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:07 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:07.914218+0000 mon.smithi061 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.61:0/2054091183' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:08.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:07 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:07.919465+0000 mon.smithi061 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.61:0/2054091183' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/mirror_snapshot_schedule"}]: dispatch 2023-12-25T21:06:08.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:07 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:07.920671+0000 mon.smithi061 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.61:0/2054091183' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/trash_purge_schedule"}]: dispatch 2023-12-25T21:06:08.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:07 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:07.925509+0000 mon.smithi061 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.61:0/2054091183' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:08.206 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:07 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:07.927434+0000 mon.smithi061 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.61:0/2054091183' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:08.384 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-25T21:06:08.764 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.762+0000 7fa385d92700 1 Processor -- start 2023-12-25T21:06:08.764 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.762+0000 7fa385d92700 1 -- start start 2023-12-25T21:06:08.764 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.762+0000 7fa385d92700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3800fa9d0 0x7fa3800fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:08.765 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.762+0000 7fa385d92700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3800fb2e0 con 0x7fa3800fa9d0 2023-12-25T21:06:08.765 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.762+0000 7fa384d90700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3800fa9d0 0x7fa3800fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:08.765 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.762+0000 7fa384d90700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3800fa9d0 0x7fa3800fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:39882/0 (socket says 172.21.15.61:39882) 2023-12-25T21:06:08.765 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.762+0000 7fa384d90700 1 -- 172.21.15.61:0/2102299789 learned_addr learned my addr 172.21.15.61:0/2102299789 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:08.765 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.762+0000 7fa384d90700 1 -- 172.21.15.61:0/2102299789 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3800fb420 con 0x7fa3800fa9d0 2023-12-25T21:06:08.765 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.763+0000 7fa384d90700 1 --2- 172.21.15.61:0/2102299789 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3800fa9d0 0x7fa3800fadf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa37400b870 tx=0x7fa374006f70).ready entity=mon.0 client_cookie=350db14c57b01f55 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:08.765 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.763+0000 7fa37f7fe700 1 -- 172.21.15.61:0/2102299789 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa37400da20 con 0x7fa3800fa9d0 2023-12-25T21:06:08.765 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.763+0000 7fa37f7fe700 1 -- 172.21.15.61:0/2102299789 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa374010470 con 0x7fa3800fa9d0 2023-12-25T21:06:08.765 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.763+0000 7fa37f7fe700 1 -- 172.21.15.61:0/2102299789 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa37400f5b0 con 0x7fa3800fa9d0 2023-12-25T21:06:08.765 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.763+0000 7fa385d92700 1 -- 172.21.15.61:0/2102299789 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3800fa9d0 msgr2=0x7fa3800fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:08.765 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.763+0000 7fa385d92700 1 --2- 172.21.15.61:0/2102299789 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3800fa9d0 0x7fa3800fadf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa37400b870 tx=0x7fa374006f70).stop 2023-12-25T21:06:08.765 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.763+0000 7fa385d92700 1 -- 172.21.15.61:0/2102299789 shutdown_connections 2023-12-25T21:06:08.766 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.763+0000 7fa385d92700 1 --2- 172.21.15.61:0/2102299789 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3800fa9d0 0x7fa3800fadf0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:08.766 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.763+0000 7fa385d92700 1 -- 172.21.15.61:0/2102299789 >> 172.21.15.61:0/2102299789 conn(0x7fa3800f6210 msgr2=0x7fa3800f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:08.766 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.764+0000 7fa385d92700 1 -- 172.21.15.61:0/2102299789 shutdown_connections 2023-12-25T21:06:08.766 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.764+0000 7fa385d92700 1 -- 172.21.15.61:0/2102299789 wait complete. 2023-12-25T21:06:08.768 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.764+0000 7fa385d92700 1 Processor -- start 2023-12-25T21:06:08.768 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.764+0000 7fa385d92700 1 -- start start 2023-12-25T21:06:08.768 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.764+0000 7fa385d92700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3800fa9d0 0x7fa380108570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:08.768 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.764+0000 7fa385d92700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa380108a60 con 0x7fa3800fa9d0 2023-12-25T21:06:08.768 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.765+0000 7fa384d90700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3800fa9d0 0x7fa380108570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:08.768 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.765+0000 7fa384d90700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3800fa9d0 0x7fa380108570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:39888/0 (socket says 172.21.15.61:39888) 2023-12-25T21:06:08.768 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.765+0000 7fa384d90700 1 -- 172.21.15.61:0/1802642576 learned_addr learned my addr 172.21.15.61:0/1802642576 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:08.768 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.765+0000 7fa384d90700 1 -- 172.21.15.61:0/1802642576 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37400b480 con 0x7fa3800fa9d0 2023-12-25T21:06:08.768 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.765+0000 7fa384d90700 1 --2- 172.21.15.61:0/1802642576 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3800fa9d0 0x7fa380108570 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fa374006800 tx=0x7fa374006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:08.769 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.765+0000 7fa37d7fa700 1 -- 172.21.15.61:0/1802642576 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa37400f820 con 0x7fa3800fa9d0 2023-12-25T21:06:08.769 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.765+0000 7fa385d92700 1 -- 172.21.15.61:0/1802642576 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa380108c60 con 0x7fa3800fa9d0 2023-12-25T21:06:08.769 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.765+0000 7fa37d7fa700 1 -- 172.21.15.61:0/1802642576 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa37400fdf0 con 0x7fa3800fa9d0 2023-12-25T21:06:08.769 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.765+0000 7fa37d7fa700 1 -- 172.21.15.61:0/1802642576 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa374019d50 con 0x7fa3800fa9d0 2023-12-25T21:06:08.769 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.765+0000 7fa385d92700 1 -- 172.21.15.61:0/1802642576 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa380109070 con 0x7fa3800fa9d0 2023-12-25T21:06:08.769 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.766+0000 7fa37d7fa700 1 -- 172.21.15.61:0/1802642576 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 2) v1 ==== 35959+0+0 (secure 0 0 0) 0x7fa374016030 con 0x7fa3800fa9d0 2023-12-25T21:06:08.769 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.766+0000 7fa385d92700 1 -- 172.21.15.61:0/1802642576 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa380045ad0 con 0x7fa3800fa9d0 2023-12-25T21:06:08.769 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.766+0000 7fa37d7fa700 1 -- 172.21.15.61:0/1802642576 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa374044040 con 0x7fa3800fa9d0 2023-12-25T21:06:08.769 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.768+0000 7fa37d7fa700 1 -- 172.21.15.61:0/1802642576 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fa37402ec90 con 0x7fa3800fa9d0 2023-12-25T21:06:08.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.838+0000 7fa385d92700 1 -- 172.21.15.61:0/1802642576 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa3800fc6e0 con 0x7fa3800fa9d0 2023-12-25T21:06:08.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.838+0000 7fa37d7fa700 1 -- 172.21.15.61:0/1802642576 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fa374014030 con 0x7fa3800fa9d0 2023-12-25T21:06:08.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: { 2023-12-25T21:06:08.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "fsid": "5532d712-a369-11ee-95a5-87774f69a715", 2023-12-25T21:06:08.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "health": { 2023-12-25T21:06:08.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-25T21:06:08.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "checks": {}, 2023-12-25T21:06:08.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "mutes": [] 2023-12-25T21:06:08.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:08.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-25T21:06:08.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum": [ 2023-12-25T21:06:08.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 0 2023-12-25T21:06:08.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-12-25T21:06:08.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-25T21:06:08.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "smithi061" 2023-12-25T21:06:08.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-12-25T21:06:08.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum_age": 7, 2023-12-25T21:06:08.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "monmap": { 2023-12-25T21:06:08.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T21:06:08.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-25T21:06:08.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-25T21:06:08.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:08.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osdmap": { 2023-12-25T21:06:08.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T21:06:08.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-25T21:06:08.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-25T21:06:08.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-25T21:06:08.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-25T21:06:08.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-25T21:06:08.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-25T21:06:08.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:08.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "pgmap": { 2023-12-25T21:06:08.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-25T21:06:08.849 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-25T21:06:08.849 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-25T21:06:08.849 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-25T21:06:08.849 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-25T21:06:08.849 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-25T21:06:08.849 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-25T21:06:08.849 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-25T21:06:08.849 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:08.849 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "fsmap": { 2023-12-25T21:06:08.849 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T21:06:08.849 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-25T21:06:08.849 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-25T21:06:08.849 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:08.849 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-25T21:06:08.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "available": false, 2023-12-25T21:06:08.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-25T21:06:08.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "modules": [ 2023-12-25T21:06:08.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "iostat", 2023-12-25T21:06:08.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "restful" 2023-12-25T21:06:08.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-12-25T21:06:08.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "services": {} 2023-12-25T21:06:08.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:08.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "servicemap": { 2023-12-25T21:06:08.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T21:06:08.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "modified": "2023-12-25T21:05:57.845222+0000", 2023-12-25T21:06:08.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "services": {} 2023-12-25T21:06:08.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:08.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-25T21:06:08.851 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: } 2023-12-25T21:06:08.851 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.839+0000 7fa385d92700 1 -- 172.21.15.61:0/1802642576 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3800fa9d0 msgr2=0x7fa380108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:08.851 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.839+0000 7fa385d92700 1 --2- 172.21.15.61:0/1802642576 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3800fa9d0 0x7fa380108570 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fa374006800 tx=0x7fa374006f70).stop 2023-12-25T21:06:08.851 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.839+0000 7fa385d92700 1 -- 172.21.15.61:0/1802642576 shutdown_connections 2023-12-25T21:06:08.851 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.839+0000 7fa385d92700 1 --2- 172.21.15.61:0/1802642576 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3800fa9d0 0x7fa380108570 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:08.851 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.839+0000 7fa385d92700 1 -- 172.21.15.61:0/1802642576 >> 172.21.15.61:0/1802642576 conn(0x7fa3800f6210 msgr2=0x7fa3800ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:08.851 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.839+0000 7fa385d92700 1 -- 172.21.15.61:0/1802642576 shutdown_connections 2023-12-25T21:06:08.851 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:08.839+0000 7fa385d92700 1 -- 172.21.15.61:0/1802642576 wait complete. 2023-12-25T21:06:09.142 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:08 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:08.839506+0000 mon.smithi061 (mon.0) 22 : audit [DBG] from='client.? 172.21.15.61:0/1802642576' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-25T21:06:09.142 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:08 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:08.895097+0000 mon.smithi061 (mon.0) 23 : cluster [DBG] mgrmap e3: smithi061.jxxeke(active, since 1.00546s) 2023-12-25T21:06:09.182 INFO:teuthology.orchestra.run.smithi061.stderr:mgr not available, waiting (3/15)... 2023-12-25T21:06:11.184 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-25T21:06:11.205 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:10 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:09.946856+0000 mon.smithi061 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi061.jxxeke(active, since 2s) 2023-12-25T21:06:11.587 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.584+0000 7fbe2beff700 1 Processor -- start 2023-12-25T21:06:11.587 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.585+0000 7fbe2beff700 1 -- start start 2023-12-25T21:06:11.587 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.585+0000 7fbe2beff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe240fab80 0x7fbe240fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:11.587 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.585+0000 7fbe2beff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe240fb490 con 0x7fbe240fab80 2023-12-25T21:06:11.587 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.585+0000 7fbe29c9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe240fab80 0x7fbe240fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:11.588 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.585+0000 7fbe29c9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe240fab80 0x7fbe240fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43774/0 (socket says 172.21.15.61:43774) 2023-12-25T21:06:11.588 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.585+0000 7fbe29c9b700 1 -- 172.21.15.61:0/3106367106 learned_addr learned my addr 172.21.15.61:0/3106367106 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:11.588 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.585+0000 7fbe29c9b700 1 -- 172.21.15.61:0/3106367106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe240fb5d0 con 0x7fbe240fab80 2023-12-25T21:06:11.588 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.585+0000 7fbe29c9b700 1 --2- 172.21.15.61:0/3106367106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe240fab80 0x7fbe240fafa0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fbe2000b5f0 tx=0x7fbe20006ee0).ready entity=mon.0 client_cookie=aafe4c178e74c1e1 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:11.588 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.586+0000 7fbe28c99700 1 -- 172.21.15.61:0/3106367106 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe2000d600 con 0x7fbe240fab80 2023-12-25T21:06:11.588 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.586+0000 7fbe28c99700 1 -- 172.21.15.61:0/3106367106 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbe2000dbd0 con 0x7fbe240fab80 2023-12-25T21:06:11.588 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.586+0000 7fbe28c99700 1 -- 172.21.15.61:0/3106367106 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe20011c00 con 0x7fbe240fab80 2023-12-25T21:06:11.588 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.586+0000 7fbe2beff700 1 -- 172.21.15.61:0/3106367106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe240fab80 msgr2=0x7fbe240fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:11.588 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.586+0000 7fbe2beff700 1 --2- 172.21.15.61:0/3106367106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe240fab80 0x7fbe240fafa0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fbe2000b5f0 tx=0x7fbe20006ee0).stop 2023-12-25T21:06:11.588 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.586+0000 7fbe2beff700 1 -- 172.21.15.61:0/3106367106 shutdown_connections 2023-12-25T21:06:11.588 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.586+0000 7fbe2beff700 1 --2- 172.21.15.61:0/3106367106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe240fab80 0x7fbe240fafa0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:11.588 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.586+0000 7fbe2beff700 1 -- 172.21.15.61:0/3106367106 >> 172.21.15.61:0/3106367106 conn(0x7fbe240f63c0 msgr2=0x7fbe240f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:11.588 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.587+0000 7fbe2beff700 1 -- 172.21.15.61:0/3106367106 shutdown_connections 2023-12-25T21:06:11.589 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.587+0000 7fbe2beff700 1 -- 172.21.15.61:0/3106367106 wait complete. 2023-12-25T21:06:11.591 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.587+0000 7fbe2beff700 1 Processor -- start 2023-12-25T21:06:11.591 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.587+0000 7fbe2beff700 1 -- start start 2023-12-25T21:06:11.591 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.587+0000 7fbe2beff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe240fab80 0x7fbe24100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:11.591 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.587+0000 7fbe2beff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe240ff290 con 0x7fbe240fab80 2023-12-25T21:06:11.591 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.587+0000 7fbe29c9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe240fab80 0x7fbe24100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:11.591 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.588+0000 7fbe29c9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe240fab80 0x7fbe24100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43784/0 (socket says 172.21.15.61:43784) 2023-12-25T21:06:11.591 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.588+0000 7fbe29c9b700 1 -- 172.21.15.61:0/1269625451 learned_addr learned my addr 172.21.15.61:0/1269625451 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:11.591 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.588+0000 7fbe29c9b700 1 -- 172.21.15.61:0/1269625451 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2000b200 con 0x7fbe240fab80 2023-12-25T21:06:11.591 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.588+0000 7fbe29c9b700 1 --2- 172.21.15.61:0/1269625451 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe240fab80 0x7fbe24100e10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fbe2000e040 tx=0x7fbe20006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:11.591 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.588+0000 7fbe127fc700 1 -- 172.21.15.61:0/1269625451 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe2001b410 con 0x7fbe240fab80 2023-12-25T21:06:11.591 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.588+0000 7fbe127fc700 1 -- 172.21.15.61:0/1269625451 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbe2001b9e0 con 0x7fbe240fab80 2023-12-25T21:06:11.592 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.588+0000 7fbe2beff700 1 -- 172.21.15.61:0/1269625451 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe240ff430 con 0x7fbe240fab80 2023-12-25T21:06:11.592 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.588+0000 7fbe2beff700 1 -- 172.21.15.61:0/1269625451 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe240ff840 con 0x7fbe240fab80 2023-12-25T21:06:11.592 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.589+0000 7fbe127fc700 1 -- 172.21.15.61:0/1269625451 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe2001a820 con 0x7fbe240fab80 2023-12-25T21:06:11.592 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.589+0000 7fbe127fc700 1 -- 172.21.15.61:0/1269625451 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7fbe2000f070 con 0x7fbe240fab80 2023-12-25T21:06:11.592 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.589+0000 7fbe127fc700 1 --2- 172.21.15.61:0/1269625451 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7fbe0c02fd00 0x7fbe0c0321c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:11.592 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.589+0000 7fbe127fc700 1 -- 172.21.15.61:0/1269625451 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fbe20042050 con 0x7fbe240fab80 2023-12-25T21:06:11.592 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.589+0000 7fbe2949a700 1 --2- 172.21.15.61:0/1269625451 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7fbe0c02fd00 0x7fbe0c0321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:11.592 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.589+0000 7fbe2beff700 1 -- 172.21.15.61:0/1269625451 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe04004fa0 con 0x7fbe240fab80 2023-12-25T21:06:11.592 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.590+0000 7fbe2949a700 1 --2- 172.21.15.61:0/1269625451 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7fbe0c02fd00 0x7fbe0c0321c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fbe140096a0 tx=0x7fbe14006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:11.594 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.594+0000 7fbe127fc700 1 -- 172.21.15.61:0/1269625451 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe20016070 con 0x7fbe240fab80 2023-12-25T21:06:11.817 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.814+0000 7fbe2beff700 1 -- 172.21.15.61:0/1269625451 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbe04005670 con 0x7fbe240fab80 2023-12-25T21:06:11.818 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.814+0000 7fbe127fc700 1 -- 172.21.15.61:0/1269625451 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fbe20025460 con 0x7fbe240fab80 2023-12-25T21:06:11.818 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.818 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: { 2023-12-25T21:06:11.818 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "fsid": "5532d712-a369-11ee-95a5-87774f69a715", 2023-12-25T21:06:11.818 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "health": { 2023-12-25T21:06:11.818 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-25T21:06:11.818 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "checks": {}, 2023-12-25T21:06:11.818 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "mutes": [] 2023-12-25T21:06:11.818 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:11.818 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-25T21:06:11.818 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum": [ 2023-12-25T21:06:11.819 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 0 2023-12-25T21:06:11.819 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-12-25T21:06:11.819 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-25T21:06:11.819 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "smithi061" 2023-12-25T21:06:11.819 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-12-25T21:06:11.819 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "quorum_age": 10, 2023-12-25T21:06:11.819 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "monmap": { 2023-12-25T21:06:11.819 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T21:06:11.820 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-25T21:06:11.820 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-25T21:06:11.820 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:11.820 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osdmap": { 2023-12-25T21:06:11.820 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T21:06:11.820 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-25T21:06:11.820 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-25T21:06:11.825 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-25T21:06:11.825 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-25T21:06:11.825 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-25T21:06:11.825 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-25T21:06:11.825 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:11.825 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "pgmap": { 2023-12-25T21:06:11.825 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-25T21:06:11.825 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-25T21:06:11.825 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-25T21:06:11.826 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-25T21:06:11.826 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-25T21:06:11.826 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-25T21:06:11.826 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-25T21:06:11.826 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-25T21:06:11.826 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:11.826 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "fsmap": { 2023-12-25T21:06:11.826 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T21:06:11.826 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-25T21:06:11.826 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-25T21:06:11.826 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:11.826 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-25T21:06:11.827 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "available": true, 2023-12-25T21:06:11.827 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-25T21:06:11.827 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "modules": [ 2023-12-25T21:06:11.827 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "iostat", 2023-12-25T21:06:11.827 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "restful" 2023-12-25T21:06:11.827 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ], 2023-12-25T21:06:11.827 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "services": {} 2023-12-25T21:06:11.827 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:11.827 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "servicemap": { 2023-12-25T21:06:11.827 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-25T21:06:11.827 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "modified": "2023-12-25T21:05:57.845222+0000", 2023-12-25T21:06:11.827 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "services": {} 2023-12-25T21:06:11.827 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: }, 2023-12-25T21:06:11.827 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-25T21:06:11.828 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: } 2023-12-25T21:06:11.828 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.816+0000 7fbe2beff700 1 -- 172.21.15.61:0/1269625451 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7fbe0c02fd00 msgr2=0x7fbe0c0321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:11.828 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.816+0000 7fbe2beff700 1 --2- 172.21.15.61:0/1269625451 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7fbe0c02fd00 0x7fbe0c0321c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fbe140096a0 tx=0x7fbe14006b40).stop 2023-12-25T21:06:11.828 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.816+0000 7fbe2beff700 1 -- 172.21.15.61:0/1269625451 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe240fab80 msgr2=0x7fbe24100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:11.828 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.816+0000 7fbe2beff700 1 --2- 172.21.15.61:0/1269625451 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe240fab80 0x7fbe24100e10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fbe2000e040 tx=0x7fbe20006ee0).stop 2023-12-25T21:06:11.828 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.816+0000 7fbe2beff700 1 -- 172.21.15.61:0/1269625451 shutdown_connections 2023-12-25T21:06:11.828 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.816+0000 7fbe2beff700 1 --2- 172.21.15.61:0/1269625451 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7fbe0c02fd00 0x7fbe0c0321c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:11.828 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.816+0000 7fbe2beff700 1 --2- 172.21.15.61:0/1269625451 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe240fab80 0x7fbe24100e10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:11.828 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.816+0000 7fbe2beff700 1 -- 172.21.15.61:0/1269625451 >> 172.21.15.61:0/1269625451 conn(0x7fbe240f63c0 msgr2=0x7fbe24070050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:11.828 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.816+0000 7fbe2beff700 1 -- 172.21.15.61:0/1269625451 shutdown_connections 2023-12-25T21:06:11.828 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:11.816+0000 7fbe2beff700 1 -- 172.21.15.61:0/1269625451 wait complete. 2023-12-25T21:06:12.182 INFO:teuthology.orchestra.run.smithi061.stderr:mgr is available 2023-12-25T21:06:12.182 INFO:teuthology.orchestra.run.smithi061.stderr:Enabling cephadm module... 2023-12-25T21:06:12.182 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2023-12-25T21:06:12.455 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:12 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:11.815678+0000 mon.smithi061 (mon.0) 25 : audit [DBG] from='client.? 172.21.15.61:0/1269625451' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-25T21:06:12.563 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.556+0000 7f9be16e7700 1 Processor -- start 2023-12-25T21:06:12.563 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.556+0000 7f9be16e7700 1 -- start start 2023-12-25T21:06:12.563 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.556+0000 7f9be16e7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9bdc0704a0 0x7f9bdc0fe770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:12.563 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.556+0000 7f9be16e7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc0fec60 con 0x7f9bdc0704a0 2023-12-25T21:06:12.564 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.556+0000 7f9bdaffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9bdc0704a0 0x7f9bdc0fe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:12.564 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.556+0000 7f9bdaffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9bdc0704a0 0x7f9bdc0fe770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43798/0 (socket says 172.21.15.61:43798) 2023-12-25T21:06:12.564 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.556+0000 7f9bdaffd700 1 -- 172.21.15.61:0/1778959177 learned_addr learned my addr 172.21.15.61:0/1778959177 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:12.564 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.556+0000 7f9bdaffd700 1 -- 172.21.15.61:0/1778959177 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bdc0feda0 con 0x7f9bdc0704a0 2023-12-25T21:06:12.564 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.556+0000 7f9bdaffd700 1 --2- 172.21.15.61:0/1778959177 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9bdc0704a0 0x7f9bdc0fe770 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f9bcc00b5f0 tx=0x7f9bcc006ee0).ready entity=mon.0 client_cookie=353a2db3a834ca48 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:12.564 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.557+0000 7f9bd9ffb700 1 -- 172.21.15.61:0/1778959177 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bcc00d600 con 0x7f9bdc0704a0 2023-12-25T21:06:12.564 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.557+0000 7f9bd9ffb700 1 -- 172.21.15.61:0/1778959177 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9bcc00dbd0 con 0x7f9bdc0704a0 2023-12-25T21:06:12.564 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.557+0000 7f9bd9ffb700 1 -- 172.21.15.61:0/1778959177 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bcc011c00 con 0x7f9bdc0704a0 2023-12-25T21:06:12.565 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.557+0000 7f9be16e7700 1 -- 172.21.15.61:0/1778959177 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9bdc0704a0 msgr2=0x7f9bdc0fe770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:12.565 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.557+0000 7f9be16e7700 1 --2- 172.21.15.61:0/1778959177 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9bdc0704a0 0x7f9bdc0fe770 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f9bcc00b5f0 tx=0x7f9bcc006ee0).stop 2023-12-25T21:06:12.565 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.557+0000 7f9be16e7700 1 -- 172.21.15.61:0/1778959177 shutdown_connections 2023-12-25T21:06:12.565 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.557+0000 7f9be16e7700 1 --2- 172.21.15.61:0/1778959177 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9bdc0704a0 0x7f9bdc0fe770 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:12.565 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.557+0000 7f9be16e7700 1 -- 172.21.15.61:0/1778959177 >> 172.21.15.61:0/1778959177 conn(0x7f9bdc0f6250 msgr2=0x7f9bdc0f8670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:12.566 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.557+0000 7f9be16e7700 1 -- 172.21.15.61:0/1778959177 shutdown_connections 2023-12-25T21:06:12.566 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.557+0000 7f9be16e7700 1 -- 172.21.15.61:0/1778959177 wait complete. 2023-12-25T21:06:12.566 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.558+0000 7f9be16e7700 1 Processor -- start 2023-12-25T21:06:12.566 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.558+0000 7f9be16e7700 1 -- start start 2023-12-25T21:06:12.566 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.558+0000 7f9be16e7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9bdc0704a0 0x7f9bdc10a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:12.566 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.558+0000 7f9be16e7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc10adc0 con 0x7f9bdc0704a0 2023-12-25T21:06:12.566 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.558+0000 7f9bdaffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9bdc0704a0 0x7f9bdc10a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:12.567 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.558+0000 7f9bdaffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9bdc0704a0 0x7f9bdc10a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43804/0 (socket says 172.21.15.61:43804) 2023-12-25T21:06:12.567 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.558+0000 7f9bdaffd700 1 -- 172.21.15.61:0/3907687712 learned_addr learned my addr 172.21.15.61:0/3907687712 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:12.567 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.558+0000 7f9bdaffd700 1 -- 172.21.15.61:0/3907687712 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc00b200 con 0x7f9bdc0704a0 2023-12-25T21:06:12.567 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.559+0000 7f9bdaffd700 1 --2- 172.21.15.61:0/3907687712 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9bdc0704a0 0x7f9bdc10a8d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f9bcc00b1d0 tx=0x7f9bcc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:12.567 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.559+0000 7f9bc37fe700 1 -- 172.21.15.61:0/3907687712 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bcc011a20 con 0x7f9bdc0704a0 2023-12-25T21:06:12.567 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.559+0000 7f9bc37fe700 1 -- 172.21.15.61:0/3907687712 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9bcc01b880 con 0x7f9bdc0704a0 2023-12-25T21:06:12.568 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.559+0000 7f9be16e7700 1 -- 172.21.15.61:0/3907687712 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bdc10afc0 con 0x7f9bdc0704a0 2023-12-25T21:06:12.568 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.559+0000 7f9bc37fe700 1 -- 172.21.15.61:0/3907687712 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bcc01a700 con 0x7f9bdc0704a0 2023-12-25T21:06:12.568 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.559+0000 7f9be16e7700 1 -- 172.21.15.61:0/3907687712 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bdc10b3d0 con 0x7f9bdc0704a0 2023-12-25T21:06:12.568 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.559+0000 7f9bc37fe700 1 -- 172.21.15.61:0/3907687712 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f9bcc00f070 con 0x7f9bdc0704a0 2023-12-25T21:06:12.568 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.560+0000 7f9bc37fe700 1 --2- 172.21.15.61:0/3907687712 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7f9bc802fd50 0x7f9bc8032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:12.569 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.560+0000 7f9bc37fe700 1 -- 172.21.15.61:0/3907687712 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9bcc041f20 con 0x7f9bdc0704a0 2023-12-25T21:06:12.569 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.560+0000 7f9bda7fc700 1 --2- 172.21.15.61:0/3907687712 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7f9bc802fd50 0x7f9bc8032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:12.569 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.560+0000 7f9be16e7700 1 -- 172.21.15.61:0/3907687712 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bdc059100 con 0x7f9bdc0704a0 2023-12-25T21:06:12.569 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.560+0000 7f9bda7fc700 1 --2- 172.21.15.61:0/3907687712 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7f9bc802fd50 0x7f9bc8032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9bd00096a0 tx=0x7f9bd0006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:12.570 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.564+0000 7f9bc37fe700 1 -- 172.21.15.61:0/3907687712 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9bcc022900 con 0x7f9bdc0704a0 2023-12-25T21:06:12.759 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:12.758+0000 7f9be16e7700 1 -- 172.21.15.61:0/3907687712 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f9bdc045ad0 con 0x7f9bdc0704a0 2023-12-25T21:06:13.272 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:13.270+0000 7f9bc37fe700 1 -- 172.21.15.61:0/3907687712 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f9bcc01a860 con 0x7f9bdc0704a0 2023-12-25T21:06:13.272 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:13.272+0000 7f9bc37fe700 1 -- 172.21.15.61:0/3907687712 <== mon.0 v2:172.21.15.61:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f9bcc0417f0 con 0x7f9bdc0704a0 2023-12-25T21:06:13.275 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:13.274+0000 7f9be16e7700 1 -- 172.21.15.61:0/3907687712 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7f9bc802fd50 msgr2=0x7f9bc8032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:13.275 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:13.274+0000 7f9be16e7700 1 --2- 172.21.15.61:0/3907687712 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7f9bc802fd50 0x7f9bc8032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9bd00096a0 tx=0x7f9bd0006b40).stop 2023-12-25T21:06:13.275 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:13.274+0000 7f9be16e7700 1 -- 172.21.15.61:0/3907687712 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9bdc0704a0 msgr2=0x7f9bdc10a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:13.275 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:13.274+0000 7f9be16e7700 1 --2- 172.21.15.61:0/3907687712 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9bdc0704a0 0x7f9bdc10a8d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f9bcc00b1d0 tx=0x7f9bcc006ee0).stop 2023-12-25T21:06:13.276 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:13.274+0000 7f9be16e7700 1 -- 172.21.15.61:0/3907687712 shutdown_connections 2023-12-25T21:06:13.276 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:13.274+0000 7f9be16e7700 1 --2- 172.21.15.61:0/3907687712 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7f9bc802fd50 0x7f9bc8032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:13.276 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:13.274+0000 7f9be16e7700 1 --2- 172.21.15.61:0/3907687712 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9bdc0704a0 0x7f9bdc10a8d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:13.276 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:13.274+0000 7f9be16e7700 1 -- 172.21.15.61:0/3907687712 >> 172.21.15.61:0/3907687712 conn(0x7f9bdc0f6250 msgr2=0x7f9bdc0f7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:13.276 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:13.276+0000 7f9be16e7700 1 -- 172.21.15.61:0/3907687712 shutdown_connections 2023-12-25T21:06:13.276 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:13.276+0000 7f9be16e7700 1 -- 172.21.15.61:0/3907687712 wait complete. 2023-12-25T21:06:13.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:13 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:12.760048+0000 mon.smithi061 (mon.0) 26 : audit [INF] from='client.? 172.21.15.61:0/3907687712' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-12-25T21:06:14.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:14 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:13.272960+0000 mon.smithi061 (mon.0) 27 : audit [INF] from='client.? 172.21.15.61:0/3907687712' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-12-25T21:06:14.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:14 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:13.273023+0000 mon.smithi061 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi061.jxxeke(active, since 5s) 2023-12-25T21:06:14.922 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-12-25T21:06:15.548 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.547+0000 7fd06efa2700 1 Processor -- start 2023-12-25T21:06:15.549 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.547+0000 7fd06efa2700 1 -- start start 2023-12-25T21:06:15.549 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.547+0000 7fd06efa2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0680703b0 0x7fd068100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:15.549 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.547+0000 7fd06efa2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0680fc5a0 con 0x7fd0680703b0 2023-12-25T21:06:15.549 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.548+0000 7fd06dfa0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0680703b0 0x7fd068100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:15.549 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.548+0000 7fd06dfa0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0680703b0 0x7fd068100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43838/0 (socket says 172.21.15.61:43838) 2023-12-25T21:06:15.549 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.548+0000 7fd06dfa0700 1 -- 172.21.15.61:0/3320787148 learned_addr learned my addr 172.21.15.61:0/3320787148 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:15.549 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.548+0000 7fd06dfa0700 1 -- 172.21.15.61:0/3320787148 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0681010a0 con 0x7fd0680703b0 2023-12-25T21:06:15.551 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.548+0000 7fd06dfa0700 1 --2- 172.21.15.61:0/3320787148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0680703b0 0x7fd068100bb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fd06400b5f0 tx=0x7fd064006ee0).ready entity=mon.0 client_cookie=c9d17ced897f04a8 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:15.551 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.548+0000 7fd06cf9e700 1 -- 172.21.15.61:0/3320787148 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd06400d600 con 0x7fd0680703b0 2023-12-25T21:06:15.551 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.548+0000 7fd06cf9e700 1 -- 172.21.15.61:0/3320787148 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd06400dbd0 con 0x7fd0680703b0 2023-12-25T21:06:15.551 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.548+0000 7fd06cf9e700 1 -- 172.21.15.61:0/3320787148 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd064011c00 con 0x7fd0680703b0 2023-12-25T21:06:15.551 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.549+0000 7fd06efa2700 1 -- 172.21.15.61:0/3320787148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0680703b0 msgr2=0x7fd068100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:15.551 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.549+0000 7fd06efa2700 1 --2- 172.21.15.61:0/3320787148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0680703b0 0x7fd068100bb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fd06400b5f0 tx=0x7fd064006ee0).stop 2023-12-25T21:06:15.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.549+0000 7fd06efa2700 1 -- 172.21.15.61:0/3320787148 shutdown_connections 2023-12-25T21:06:15.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.549+0000 7fd06efa2700 1 --2- 172.21.15.61:0/3320787148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0680703b0 0x7fd068100bb0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:15.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.549+0000 7fd06efa2700 1 -- 172.21.15.61:0/3320787148 >> 172.21.15.61:0/3320787148 conn(0x7fd0680f6210 msgr2=0x7fd0680f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:15.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.549+0000 7fd06efa2700 1 -- 172.21.15.61:0/3320787148 shutdown_connections 2023-12-25T21:06:15.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.549+0000 7fd06efa2700 1 -- 172.21.15.61:0/3320787148 wait complete. 2023-12-25T21:06:15.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.550+0000 7fd06efa2700 1 Processor -- start 2023-12-25T21:06:15.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.550+0000 7fd06efa2700 1 -- start start 2023-12-25T21:06:15.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.550+0000 7fd06efa2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0680703b0 0x7fd06810ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:15.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.550+0000 7fd06efa2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06810cf60 con 0x7fd0680703b0 2023-12-25T21:06:15.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.550+0000 7fd06dfa0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0680703b0 0x7fd06810ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:15.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.550+0000 7fd06dfa0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0680703b0 0x7fd06810ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43846/0 (socket says 172.21.15.61:43846) 2023-12-25T21:06:15.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.550+0000 7fd06dfa0700 1 -- 172.21.15.61:0/3374954502 learned_addr learned my addr 172.21.15.61:0/3374954502 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:15.553 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.551+0000 7fd06dfa0700 1 -- 172.21.15.61:0/3374954502 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd06400b200 con 0x7fd0680703b0 2023-12-25T21:06:15.553 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.551+0000 7fd06dfa0700 1 --2- 172.21.15.61:0/3374954502 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0680703b0 0x7fd06810ca70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fd06400e040 tx=0x7fd064006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:15.554 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.551+0000 7fd0567fc700 1 -- 172.21.15.61:0/3374954502 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd06400d9f0 con 0x7fd0680703b0 2023-12-25T21:06:15.554 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.551+0000 7fd0567fc700 1 -- 172.21.15.61:0/3374954502 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd06401a9f0 con 0x7fd0680703b0 2023-12-25T21:06:15.554 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.551+0000 7fd06efa2700 1 -- 172.21.15.61:0/3374954502 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd06810d160 con 0x7fd0680703b0 2023-12-25T21:06:15.554 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.551+0000 7fd0567fc700 1 -- 172.21.15.61:0/3374954502 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd06401b8c0 con 0x7fd0680703b0 2023-12-25T21:06:15.554 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.551+0000 7fd06efa2700 1 -- 172.21.15.61:0/3374954502 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd06810d570 con 0x7fd0680703b0 2023-12-25T21:06:15.554 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.552+0000 7fd0567fc700 1 -- 172.21.15.61:0/3374954502 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fd06400f070 con 0x7fd0680703b0 2023-12-25T21:06:15.554 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.552+0000 7fd0567fc700 1 --2- 172.21.15.61:0/3374954502 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7fd05002fdb0 0x7fd050032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:15.554 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.552+0000 7fd0567fc700 1 -- 172.21.15.61:0/3374954502 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd064042030 con 0x7fd0680703b0 2023-12-25T21:06:15.555 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.552+0000 7fd06d79f700 1 -- 172.21.15.61:0/3374954502 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7fd05002fdb0 msgr2=0x7fd050032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/3955759379 2023-12-25T21:06:15.555 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.552+0000 7fd06d79f700 1 --2- 172.21.15.61:0/3374954502 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7fd05002fdb0 0x7fd050032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-25T21:06:15.555 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.552+0000 7fd06efa2700 1 -- 172.21.15.61:0/3374954502 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd068045ad0 con 0x7fd0680703b0 2023-12-25T21:06:15.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.557+0000 7fd0567fc700 1 -- 172.21.15.61:0/3374954502 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd064016070 con 0x7fd0680703b0 2023-12-25T21:06:15.754 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.753+0000 7fd06d79f700 1 -- 172.21.15.61:0/3374954502 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7fd05002fdb0 msgr2=0x7fd050032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/3955759379 2023-12-25T21:06:15.754 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.753+0000 7fd06d79f700 1 --2- 172.21.15.61:0/3374954502 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7fd05002fdb0 0x7fd050032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-25T21:06:15.756 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.755+0000 7fd06efa2700 1 -- 172.21.15.61:0/3374954502 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd0681068f0 con 0x7fd0680703b0 2023-12-25T21:06:15.756 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.756+0000 7fd0567fc700 1 -- 172.21.15.61:0/3374954502 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fd064025460 con 0x7fd0680703b0 2023-12-25T21:06:15.757 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: { 2023-12-25T21:06:15.757 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 5, 2023-12-25T21:06:15.757 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "available": true, 2023-12-25T21:06:15.757 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "active_name": "smithi061.jxxeke", 2023-12-25T21:06:15.757 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_standby": 0 2023-12-25T21:06:15.757 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: } 2023-12-25T21:06:15.759 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.757+0000 7fd06efa2700 1 -- 172.21.15.61:0/3374954502 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7fd05002fdb0 msgr2=0x7fd050032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-25T21:06:15.759 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.757+0000 7fd06efa2700 1 --2- 172.21.15.61:0/3374954502 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7fd05002fdb0 0x7fd050032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T21:06:15.759 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.757+0000 7fd06efa2700 1 -- 172.21.15.61:0/3374954502 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0680703b0 msgr2=0x7fd06810ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:15.759 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.757+0000 7fd06efa2700 1 --2- 172.21.15.61:0/3374954502 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0680703b0 0x7fd06810ca70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fd06400e040 tx=0x7fd064006ee0).stop 2023-12-25T21:06:15.759 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.757+0000 7fd06efa2700 1 -- 172.21.15.61:0/3374954502 shutdown_connections 2023-12-25T21:06:15.759 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.757+0000 7fd06efa2700 1 --2- 172.21.15.61:0/3374954502 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7fd05002fdb0 0x7fd050032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T21:06:15.759 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.757+0000 7fd06efa2700 1 --2- 172.21.15.61:0/3374954502 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0680703b0 0x7fd06810ca70 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:15.759 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.757+0000 7fd06efa2700 1 -- 172.21.15.61:0/3374954502 >> 172.21.15.61:0/3374954502 conn(0x7fd0680f6210 msgr2=0x7fd0680f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:15.759 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.757+0000 7fd06efa2700 1 -- 172.21.15.61:0/3374954502 shutdown_connections 2023-12-25T21:06:15.759 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:15.757+0000 7fd06efa2700 1 -- 172.21.15.61:0/3374954502 wait complete. 2023-12-25T21:06:16.099 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:15 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:15.756811+0000 mon.smithi061 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.61:0/3374954502' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-25T21:06:16.122 INFO:teuthology.orchestra.run.smithi061.stderr:Waiting for the mgr to restart... 2023-12-25T21:06:16.123 INFO:teuthology.orchestra.run.smithi061.stderr:Waiting for mgr epoch 5... 2023-12-25T21:06:16.123 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-12-25T21:06:16.490 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.488+0000 7f6e0f2c0700 1 Processor -- start 2023-12-25T21:06:16.490 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.489+0000 7f6e0f2c0700 1 -- start start 2023-12-25T21:06:16.490 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.489+0000 7f6e0f2c0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e08072490 0x7f6e08070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:16.490 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.489+0000 7f6e0f2c0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e080728b0 con 0x7f6e08072490 2023-12-25T21:06:16.490 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.489+0000 7f6e0d05c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e08072490 0x7f6e08070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:16.490 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.489+0000 7f6e0d05c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e08072490 0x7f6e08070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43848/0 (socket says 172.21.15.61:43848) 2023-12-25T21:06:16.490 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.489+0000 7f6e0d05c700 1 -- 172.21.15.61:0/2682129325 learned_addr learned my addr 172.21.15.61:0/2682129325 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:16.490 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.489+0000 7f6e0d05c700 1 -- 172.21.15.61:0/2682129325 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e08070f20 con 0x7f6e08072490 2023-12-25T21:06:16.495 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.489+0000 7f6e0d05c700 1 --2- 172.21.15.61:0/2682129325 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e08072490 0x7f6e08070a30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f6e0400b5f0 tx=0x7f6e04006ee0).ready entity=mon.0 client_cookie=53771afe133f23a2 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:16.495 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.490+0000 7f6df7fff700 1 -- 172.21.15.61:0/2682129325 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e0400d600 con 0x7f6e08072490 2023-12-25T21:06:16.495 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.490+0000 7f6df7fff700 1 -- 172.21.15.61:0/2682129325 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6e0400dbd0 con 0x7f6e08072490 2023-12-25T21:06:16.495 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.490+0000 7f6df7fff700 1 -- 172.21.15.61:0/2682129325 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e04011c00 con 0x7f6e08072490 2023-12-25T21:06:16.495 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.490+0000 7f6e0f2c0700 1 -- 172.21.15.61:0/2682129325 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e08072490 msgr2=0x7f6e08070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:16.495 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.490+0000 7f6e0f2c0700 1 --2- 172.21.15.61:0/2682129325 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e08072490 0x7f6e08070a30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f6e0400b5f0 tx=0x7f6e04006ee0).stop 2023-12-25T21:06:16.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.490+0000 7f6e0f2c0700 1 -- 172.21.15.61:0/2682129325 shutdown_connections 2023-12-25T21:06:16.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.490+0000 7f6e0f2c0700 1 --2- 172.21.15.61:0/2682129325 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e08072490 0x7f6e08070a30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:16.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.490+0000 7f6e0f2c0700 1 -- 172.21.15.61:0/2682129325 >> 172.21.15.61:0/2682129325 conn(0x7f6e080f74c0 msgr2=0x7f6e080f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:16.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.490+0000 7f6e0f2c0700 1 -- 172.21.15.61:0/2682129325 shutdown_connections 2023-12-25T21:06:16.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.490+0000 7f6e0f2c0700 1 -- 172.21.15.61:0/2682129325 wait complete. 2023-12-25T21:06:16.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.491+0000 7f6e0f2c0700 1 Processor -- start 2023-12-25T21:06:16.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.491+0000 7f6e0f2c0700 1 -- start start 2023-12-25T21:06:16.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.491+0000 7f6e0f2c0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e08072490 0x7f6e0810cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:16.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.491+0000 7f6e0f2c0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0810d250 con 0x7f6e08072490 2023-12-25T21:06:16.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.491+0000 7f6e0d05c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e08072490 0x7f6e0810cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:16.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.491+0000 7f6e0d05c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e08072490 0x7f6e0810cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43864/0 (socket says 172.21.15.61:43864) 2023-12-25T21:06:16.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.491+0000 7f6e0d05c700 1 -- 172.21.15.61:0/1590274813 learned_addr learned my addr 172.21.15.61:0/1590274813 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:16.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.492+0000 7f6e0d05c700 1 -- 172.21.15.61:0/1590274813 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0400b200 con 0x7f6e08072490 2023-12-25T21:06:16.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.492+0000 7f6e0d05c700 1 --2- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e08072490 0x7f6e0810cd60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f6e04006c70 tx=0x7f6e04006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:16.497 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.492+0000 7f6df5ffb700 1 -- 172.21.15.61:0/1590274813 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e0400da30 con 0x7f6e08072490 2023-12-25T21:06:16.497 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.492+0000 7f6df5ffb700 1 -- 172.21.15.61:0/1590274813 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6e04011690 con 0x7f6e08072490 2023-12-25T21:06:16.497 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.492+0000 7f6e0f2c0700 1 -- 172.21.15.61:0/1590274813 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0810d450 con 0x7f6e08072490 2023-12-25T21:06:16.497 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.492+0000 7f6df5ffb700 1 -- 172.21.15.61:0/1590274813 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e0401b7b0 con 0x7f6e08072490 2023-12-25T21:06:16.497 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.492+0000 7f6e0f2c0700 1 -- 172.21.15.61:0/1590274813 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0810d860 con 0x7f6e08072490 2023-12-25T21:06:16.497 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.493+0000 7f6df5ffb700 1 -- 172.21.15.61:0/1590274813 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f6e0400f070 con 0x7f6e08072490 2023-12-25T21:06:16.497 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.493+0000 7f6df5ffb700 1 --2- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7f6df002fdb0 0x7f6df0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:16.497 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.493+0000 7f6df5ffb700 1 -- 172.21.15.61:0/1590274813 --> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6df0032930 con 0x7f6df002fdb0 2023-12-25T21:06:16.497 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.493+0000 7f6df5ffb700 1 -- 172.21.15.61:0/1590274813 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6e040410b0 con 0x7f6e08072490 2023-12-25T21:06:16.497 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.493+0000 7f6e0c85b700 1 -- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7f6df002fdb0 msgr2=0x7f6df0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/3955759379 2023-12-25T21:06:16.497 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.493+0000 7f6e0c85b700 1 --2- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7f6df002fdb0 0x7f6df0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-25T21:06:16.694 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.694+0000 7f6e0c85b700 1 -- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7f6df002fdb0 msgr2=0x7f6df0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/3955759379 2023-12-25T21:06:16.695 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:16.694+0000 7f6e0c85b700 1 --2- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7f6df002fdb0 0x7f6df0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-25T21:06:17.095 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:17.095+0000 7f6e0c85b700 1 -- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7f6df002fdb0 msgr2=0x7f6df0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/3955759379 2023-12-25T21:06:17.096 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:17.095+0000 7f6e0c85b700 1 --2- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7f6df002fdb0 0x7f6df0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-25T21:06:17.897 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:17.896+0000 7f6e0c85b700 1 -- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7f6df002fdb0 msgr2=0x7f6df0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/3955759379 2023-12-25T21:06:17.897 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:17.896+0000 7f6e0c85b700 1 --2- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7f6df002fdb0 0x7f6df0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-12-25T21:06:18.496 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:18.495+0000 7f6df5ffb700 1 -- 172.21.15.61:0/1590274813 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f6e0401ab80 con 0x7f6e08072490 2023-12-25T21:06:18.497 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:18.495+0000 7f6df5ffb700 1 -- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7f6df002fdb0 msgr2=0x7f6df0032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-25T21:06:18.497 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:18.495+0000 7f6df5ffb700 1 --2- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7f6df002fdb0 0x7f6df0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T21:06:19.499 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:19.497+0000 7f6df5ffb700 1 -- 172.21.15.61:0/1590274813 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f6e0401bba0 con 0x7f6e08072490 2023-12-25T21:06:19.499 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:19.497+0000 7f6df5ffb700 1 --2- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f6df0033880 0x7f6df0035c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:19.499 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:19.497+0000 7f6df5ffb700 1 -- 172.21.15.61:0/1590274813 --> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6df0032930 con 0x7f6df0033880 2023-12-25T21:06:19.499 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:19.497+0000 7f6e0c85b700 1 --2- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f6df0033880 0x7f6df0035c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:19.499 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:19.498+0000 7f6e0c85b700 1 --2- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f6df0033880 0x7f6df0035c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f6df80036d0 tx=0x7f6df8006d20).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:19.500 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:19.499+0000 7f6df5ffb700 1 -- 172.21.15.61:0/1590274813 <== mgr.14116 v2:172.21.15.61:6800/19001851 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f6df0032930 con 0x7f6df0033880 2023-12-25T21:06:19.506 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:19.505+0000 7f6e0f2c0700 1 -- 172.21.15.61:0/1590274813 --> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f6e08059100 con 0x7f6df0033880 2023-12-25T21:06:19.506 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:19.506+0000 7f6df5ffb700 1 -- 172.21.15.61:0/1590274813 <== mgr.14116 v2:172.21.15.61:6800/19001851 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f6e08059100 con 0x7f6df0033880 2023-12-25T21:06:19.507 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: { 2023-12-25T21:06:19.507 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2023-12-25T21:06:19.507 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "initialized": true 2023-12-25T21:06:19.507 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: } 2023-12-25T21:06:19.509 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:19.506+0000 7f6e0f2c0700 1 -- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f6df0033880 msgr2=0x7f6df0035c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:19.509 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:19.506+0000 7f6e0f2c0700 1 --2- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f6df0033880 0x7f6df0035c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f6df80036d0 tx=0x7f6df8006d20).stop 2023-12-25T21:06:19.509 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:19.506+0000 7f6e0f2c0700 1 -- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e08072490 msgr2=0x7f6e0810cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:19.509 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:19.506+0000 7f6e0f2c0700 1 --2- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e08072490 0x7f6e0810cd60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f6e04006c70 tx=0x7f6e04006ee0).stop 2023-12-25T21:06:19.509 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:19.506+0000 7f6e0f2c0700 1 -- 172.21.15.61:0/1590274813 shutdown_connections 2023-12-25T21:06:19.509 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:19.506+0000 7f6e0f2c0700 1 --2- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f6df0033880 0x7f6df0035c70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:19.509 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:19.506+0000 7f6e0f2c0700 1 --2- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e08072490 0x7f6e0810cd60 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:19.510 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:19.506+0000 7f6e0f2c0700 1 --2- 172.21.15.61:0/1590274813 >> [v2:172.21.15.61:6800/3955759379,v1:172.21.15.61:6801/3955759379] conn(0x7f6df002fdb0 0x7f6df0032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T21:06:19.510 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:19.506+0000 7f6e0f2c0700 1 -- 172.21.15.61:0/1590274813 >> 172.21.15.61:0/1590274813 conn(0x7f6e080f74c0 msgr2=0x7f6e080f9080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:19.510 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:19.506+0000 7f6e0f2c0700 1 -- 172.21.15.61:0/1590274813 shutdown_connections 2023-12-25T21:06:19.510 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:19.506+0000 7f6e0f2c0700 1 -- 172.21.15.61:0/1590274813 wait complete. 2023-12-25T21:06:19.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:18.443957+0000 mon.smithi061 (mon.0) 30 : cluster [INF] Active manager daemon smithi061.jxxeke restarted 2023-12-25T21:06:19.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:18.444418+0000 mon.smithi061 (mon.0) 31 : cluster [INF] Activating manager daemon smithi061.jxxeke 2023-12-25T21:06:19.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:18.446609+0000 mon.smithi061 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-12-25T21:06:19.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:18.497725+0000 mon.smithi061 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi061.jxxeke(active, starting, since 0.053409s) 2023-12-25T21:06:19.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:18.498648+0000 mon.smithi061 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "mon metadata", "id": "smithi061"}]: dispatch 2023-12-25T21:06:19.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:18.499255+0000 mon.smithi061 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "mgr metadata", "who": "smithi061.jxxeke", "id": "smithi061.jxxeke"}]: dispatch 2023-12-25T21:06:19.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:18.499969+0000 mon.smithi061 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-25T21:06:19.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:18.500195+0000 mon.smithi061 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-25T21:06:19.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:18.500424+0000 mon.smithi061 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-25T21:06:19.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:18.514502+0000 mon.smithi061 (mon.0) 39 : cluster [INF] Manager daemon smithi061.jxxeke is now available 2023-12-25T21:06:19.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:18.528037+0000 mon.smithi061 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:19.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:18.530914+0000 mon.smithi061 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:19.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:18.533519+0000 mon.smithi061 (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:19.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:18.627242+0000 mon.smithi061 (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:19.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:18.627965+0000 mon.smithi061 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-25T21:06:19.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:18.629834+0000 mon.smithi061 (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-25T21:06:19.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:18.631427+0000 mon.smithi061 (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-25T21:06:19.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:18.641193+0000 mon.smithi061 (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/mirror_snapshot_schedule"}]: dispatch 2023-12-25T21:06:19.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:18.642361+0000 mon.smithi061 (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/trash_purge_schedule"}]: dispatch 2023-12-25T21:06:19.856 INFO:teuthology.orchestra.run.smithi061.stderr:mgr epoch 5 is available 2023-12-25T21:06:19.856 INFO:teuthology.orchestra.run.smithi061.stderr:Setting orchestrator backend to cephadm... 2023-12-25T21:06:19.856 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2023-12-25T21:06:20.236 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.234+0000 7f43ff8ec700 1 Processor -- start 2023-12-25T21:06:20.236 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.235+0000 7f43ff8ec700 1 -- start start 2023-12-25T21:06:20.236 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.235+0000 7f43ff8ec700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43f80703f0 0x7f43f80fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:20.236 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.235+0000 7f43ff8ec700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f80fee10 con 0x7f43f80703f0 2023-12-25T21:06:20.236 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.235+0000 7f43fd688700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43f80703f0 0x7f43f80fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:20.236 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.235+0000 7f43fd688700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43f80703f0 0x7f43f80fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42876/0 (socket says 172.21.15.61:42876) 2023-12-25T21:06:20.237 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.235+0000 7f43fd688700 1 -- 172.21.15.61:0/3922333309 learned_addr learned my addr 172.21.15.61:0/3922333309 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:20.240 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.235+0000 7f43fd688700 1 -- 172.21.15.61:0/3922333309 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f80fef50 con 0x7f43f80703f0 2023-12-25T21:06:20.240 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.236+0000 7f43fd688700 1 --2- 172.21.15.61:0/3922333309 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43f80703f0 0x7f43f80fe920 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f43f400b5f0 tx=0x7f43f4006ee0).ready entity=mon.0 client_cookie=e516c912e17e53f8 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:20.240 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.236+0000 7f43ebfff700 1 -- 172.21.15.61:0/3922333309 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43f400d600 con 0x7f43f80703f0 2023-12-25T21:06:20.240 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.236+0000 7f43ebfff700 1 -- 172.21.15.61:0/3922333309 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f43f400dbd0 con 0x7f43f80703f0 2023-12-25T21:06:20.240 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.236+0000 7f43ebfff700 1 -- 172.21.15.61:0/3922333309 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43f4011c00 con 0x7f43f80703f0 2023-12-25T21:06:20.240 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.236+0000 7f43ff8ec700 1 -- 172.21.15.61:0/3922333309 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43f80703f0 msgr2=0x7f43f80fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:20.240 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.236+0000 7f43ff8ec700 1 --2- 172.21.15.61:0/3922333309 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43f80703f0 0x7f43f80fe920 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f43f400b5f0 tx=0x7f43f4006ee0).stop 2023-12-25T21:06:20.240 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.236+0000 7f43ff8ec700 1 -- 172.21.15.61:0/3922333309 shutdown_connections 2023-12-25T21:06:20.240 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.236+0000 7f43ff8ec700 1 --2- 172.21.15.61:0/3922333309 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43f80703f0 0x7f43f80fe920 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:20.240 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.236+0000 7f43ff8ec700 1 -- 172.21.15.61:0/3922333309 >> 172.21.15.61:0/3922333309 conn(0x7f43f80f63c0 msgr2=0x7f43f80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.236+0000 7f43ff8ec700 1 -- 172.21.15.61:0/3922333309 shutdown_connections 2023-12-25T21:06:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.236+0000 7f43ff8ec700 1 -- 172.21.15.61:0/3922333309 wait complete. 2023-12-25T21:06:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.237+0000 7f43ff8ec700 1 Processor -- start 2023-12-25T21:06:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.237+0000 7f43ff8ec700 1 -- start start 2023-12-25T21:06:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.237+0000 7f43ff8ec700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43f80703f0 0x7f43f80fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.237+0000 7f43ff8ec700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f80fa820 con 0x7f43f80703f0 2023-12-25T21:06:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.237+0000 7f43fd688700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43f80703f0 0x7f43f80fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.237+0000 7f43fd688700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43f80703f0 0x7f43f80fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42890/0 (socket says 172.21.15.61:42890) 2023-12-25T21:06:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.237+0000 7f43fd688700 1 -- 172.21.15.61:0/3465116623 learned_addr learned my addr 172.21.15.61:0/3465116623 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.238+0000 7f43fd688700 1 -- 172.21.15.61:0/3465116623 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f400b200 con 0x7f43f80703f0 2023-12-25T21:06:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.238+0000 7f43fd688700 1 --2- 172.21.15.61:0/3465116623 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43f80703f0 0x7f43f80fc3a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f43f400b190 tx=0x7f43f4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.238+0000 7f43e9ffb700 1 -- 172.21.15.61:0/3465116623 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43f401b410 con 0x7f43f80703f0 2023-12-25T21:06:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.238+0000 7f43e9ffb700 1 -- 172.21.15.61:0/3465116623 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f43f401b9e0 con 0x7f43f80703f0 2023-12-25T21:06:20.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.238+0000 7f43ff8ec700 1 -- 172.21.15.61:0/3465116623 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f80fa9c0 con 0x7f43f80703f0 2023-12-25T21:06:20.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.238+0000 7f43e9ffb700 1 -- 172.21.15.61:0/3465116623 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43f401a880 con 0x7f43f80703f0 2023-12-25T21:06:20.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.238+0000 7f43ff8ec700 1 -- 172.21.15.61:0/3465116623 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f80fadd0 con 0x7f43f80703f0 2023-12-25T21:06:20.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.239+0000 7f43e9ffb700 1 -- 172.21.15.61:0/3465116623 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f43f400f070 con 0x7f43f80703f0 2023-12-25T21:06:20.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.239+0000 7f43e9ffb700 1 --2- 172.21.15.61:0/3465116623 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f43e002fca0 0x7f43e0032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:20.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.239+0000 7f43e9ffb700 1 -- 172.21.15.61:0/3465116623 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f43f4041180 con 0x7f43f80703f0 2023-12-25T21:06:20.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.239+0000 7f43fce87700 1 --2- 172.21.15.61:0/3465116623 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f43e002fca0 0x7f43e0032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:20.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.239+0000 7f43ff8ec700 1 -- 172.21.15.61:0/3465116623 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f8059100 con 0x7f43f80703f0 2023-12-25T21:06:20.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.240+0000 7f43fce87700 1 --2- 172.21.15.61:0/3465116623 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f43e002fca0 0x7f43e0032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f43ec0096a0 tx=0x7f43ec006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:20.244 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.244+0000 7f43e9ffb700 1 -- 172.21.15.61:0/3465116623 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f43f4010360 con 0x7f43f80703f0 2023-12-25T21:06:20.435 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.434+0000 7f43ff8ec700 1 -- 172.21.15.61:0/3465116623 --> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f43f80ffc40 con 0x7f43e002fca0 2023-12-25T21:06:20.440 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.439+0000 7f43e9ffb700 1 -- 172.21.15.61:0/3465116623 <== mgr.14116 v2:172.21.15.61:6800/19001851 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f43f80ffc40 con 0x7f43e002fca0 2023-12-25T21:06:20.442 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.441+0000 7f43ff8ec700 1 -- 172.21.15.61:0/3465116623 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f43e002fca0 msgr2=0x7f43e0032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:20.442 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.441+0000 7f43ff8ec700 1 --2- 172.21.15.61:0/3465116623 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f43e002fca0 0x7f43e0032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f43ec0096a0 tx=0x7f43ec006b40).stop 2023-12-25T21:06:20.443 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.441+0000 7f43ff8ec700 1 -- 172.21.15.61:0/3465116623 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43f80703f0 msgr2=0x7f43f80fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:20.443 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.441+0000 7f43ff8ec700 1 --2- 172.21.15.61:0/3465116623 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43f80703f0 0x7f43f80fc3a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f43f400b190 tx=0x7f43f4006ee0).stop 2023-12-25T21:06:20.443 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.441+0000 7f43ff8ec700 1 -- 172.21.15.61:0/3465116623 shutdown_connections 2023-12-25T21:06:20.443 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.441+0000 7f43ff8ec700 1 --2- 172.21.15.61:0/3465116623 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f43e002fca0 0x7f43e0032160 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:20.443 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.441+0000 7f43ff8ec700 1 --2- 172.21.15.61:0/3465116623 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43f80703f0 0x7f43f80fc3a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:20.443 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.441+0000 7f43ff8ec700 1 -- 172.21.15.61:0/3465116623 >> 172.21.15.61:0/3465116623 conn(0x7f43f80f63c0 msgr2=0x7f43f80f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:20.443 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.441+0000 7f43ff8ec700 1 -- 172.21.15.61:0/3465116623 shutdown_connections 2023-12-25T21:06:20.443 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:20.441+0000 7f43ff8ec700 1 -- 172.21.15.61:0/3465116623 wait complete. 2023-12-25T21:06:20.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:20 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:19.500067+0000 mgr.smithi061.jxxeke (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-25T21:06:20.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:20 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:19.500628+0000 mon.smithi061 (mon.0) 49 : cluster [DBG] mgrmap e7: smithi061.jxxeke(active, since 1.05631s) 2023-12-25T21:06:20.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:20 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:19.506771+0000 mgr.smithi061.jxxeke (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-25T21:06:20.956 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:20 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:20.440291+0000 mon.smithi061 (mon.0) 50 : audit [INF] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:20.956 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:20 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:20.444094+0000 mon.smithi061 (mon.0) 51 : audit [DBG] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-25T21:06:22.414 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2023-12-25T21:06:22.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:22 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:20.436385+0000 mgr.smithi061.jxxeke (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T21:06:22.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:22 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:21.441812+0000 mon.smithi061 (mon.0) 52 : cluster [DBG] mgrmap e8: smithi061.jxxeke(active, since 2s) 2023-12-25T21:06:23.948 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.837+0000 7f8e82902700 1 Processor -- start 2023-12-25T21:06:23.948 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.837+0000 7f8e82902700 1 -- start start 2023-12-25T21:06:23.948 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.837+0000 7f8e82902700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e7c0fab80 0x7f8e7c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:23.948 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.837+0000 7f8e82902700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c0fb490 con 0x7f8e7c0fab80 2023-12-25T21:06:23.948 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.837+0000 7f8e7bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e7c0fab80 0x7f8e7c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:23.949 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.837+0000 7f8e7bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e7c0fab80 0x7f8e7c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42898/0 (socket says 172.21.15.61:42898) 2023-12-25T21:06:23.949 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.837+0000 7f8e7bfff700 1 -- 172.21.15.61:0/749430632 learned_addr learned my addr 172.21.15.61:0/749430632 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:23.949 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.838+0000 7f8e7bfff700 1 -- 172.21.15.61:0/749430632 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7c0fb5d0 con 0x7f8e7c0fab80 2023-12-25T21:06:23.949 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.838+0000 7f8e7bfff700 1 --2- 172.21.15.61:0/749430632 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e7c0fab80 0x7f8e7c0fafa0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f8e6c00b5f0 tx=0x7f8e6c006ee0).ready entity=mon.0 client_cookie=a03327707bfc752 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:23.949 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.838+0000 7f8e7affd700 1 -- 172.21.15.61:0/749430632 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e6c00d600 con 0x7f8e7c0fab80 2023-12-25T21:06:23.949 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.838+0000 7f8e7affd700 1 -- 172.21.15.61:0/749430632 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8e6c00dbd0 con 0x7f8e7c0fab80 2023-12-25T21:06:23.949 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.838+0000 7f8e7affd700 1 -- 172.21.15.61:0/749430632 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e6c011c00 con 0x7f8e7c0fab80 2023-12-25T21:06:23.949 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.838+0000 7f8e82902700 1 -- 172.21.15.61:0/749430632 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e7c0fab80 msgr2=0x7f8e7c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:23.949 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.838+0000 7f8e82902700 1 --2- 172.21.15.61:0/749430632 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e7c0fab80 0x7f8e7c0fafa0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f8e6c00b5f0 tx=0x7f8e6c006ee0).stop 2023-12-25T21:06:23.949 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.838+0000 7f8e82902700 1 -- 172.21.15.61:0/749430632 shutdown_connections 2023-12-25T21:06:23.949 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.838+0000 7f8e82902700 1 --2- 172.21.15.61:0/749430632 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e7c0fab80 0x7f8e7c0fafa0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:23.949 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.839+0000 7f8e82902700 1 -- 172.21.15.61:0/749430632 >> 172.21.15.61:0/749430632 conn(0x7f8e7c0f63c0 msgr2=0x7f8e7c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:23.950 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.839+0000 7f8e82902700 1 -- 172.21.15.61:0/749430632 shutdown_connections 2023-12-25T21:06:23.950 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.839+0000 7f8e82902700 1 -- 172.21.15.61:0/749430632 wait complete. 2023-12-25T21:06:23.950 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.839+0000 7f8e82902700 1 Processor -- start 2023-12-25T21:06:23.950 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.839+0000 7f8e82902700 1 -- start start 2023-12-25T21:06:23.950 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.839+0000 7f8e82902700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e7c0fab80 0x7f8e7c10d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:23.950 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.840+0000 7f8e82902700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c070470 con 0x7f8e7c0fab80 2023-12-25T21:06:23.950 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.840+0000 7f8e7bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e7c0fab80 0x7f8e7c10d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:23.950 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.840+0000 7f8e7bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e7c0fab80 0x7f8e7c10d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42910/0 (socket says 172.21.15.61:42910) 2023-12-25T21:06:23.950 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.840+0000 7f8e7bfff700 1 -- 172.21.15.61:0/761797240 learned_addr learned my addr 172.21.15.61:0/761797240 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:23.950 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.840+0000 7f8e7bfff700 1 -- 172.21.15.61:0/761797240 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6c00b200 con 0x7f8e7c0fab80 2023-12-25T21:06:23.950 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.840+0000 7f8e7bfff700 1 --2- 172.21.15.61:0/761797240 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e7c0fab80 0x7f8e7c10d720 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f8e6c00e040 tx=0x7f8e6c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:23.950 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.840+0000 7f8e78ff9700 1 -- 172.21.15.61:0/761797240 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e6c01b410 con 0x7f8e7c0fab80 2023-12-25T21:06:23.950 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.840+0000 7f8e78ff9700 1 -- 172.21.15.61:0/761797240 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8e6c01b9e0 con 0x7f8e7c0fab80 2023-12-25T21:06:23.950 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.840+0000 7f8e78ff9700 1 -- 172.21.15.61:0/761797240 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e6c01a820 con 0x7f8e7c0fab80 2023-12-25T21:06:23.951 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.840+0000 7f8e82902700 1 -- 172.21.15.61:0/761797240 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e7c070670 con 0x7f8e7c0fab80 2023-12-25T21:06:23.951 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.840+0000 7f8e82902700 1 -- 172.21.15.61:0/761797240 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e7c10ddd0 con 0x7f8e7c0fab80 2023-12-25T21:06:23.951 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.841+0000 7f8e78ff9700 1 -- 172.21.15.61:0/761797240 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f8e6c00f070 con 0x7f8e7c0fab80 2023-12-25T21:06:23.951 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.841+0000 7f8e78ff9700 1 --2- 172.21.15.61:0/761797240 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f8e6802fdb0 0x7f8e68032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:23.951 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.841+0000 7f8e78ff9700 1 -- 172.21.15.61:0/761797240 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f8e6c041350 con 0x7f8e7c0fab80 2023-12-25T21:06:23.951 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.841+0000 7f8e82902700 1 -- 172.21.15.61:0/761797240 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e7c045ad0 con 0x7f8e7c0fab80 2023-12-25T21:06:23.951 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.841+0000 7f8e7b7fe700 1 --2- 172.21.15.61:0/761797240 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f8e6802fdb0 0x7f8e68032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:23.951 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.842+0000 7f8e7b7fe700 1 --2- 172.21.15.61:0/761797240 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f8e6802fdb0 0x7f8e68032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f8e700096a0 tx=0x7f8e70006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:23.951 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:23.846+0000 7f8e78ff9700 1 -- 172.21.15.61:0/761797240 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8e6c016080 con 0x7f8e7c0fab80 2023-12-25T21:06:24.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:24.005+0000 7f8e82902700 1 -- 172.21.15.61:0/761797240 --> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e7c0fc9d0 con 0x7f8e6802fdb0 2023-12-25T21:06:24.006 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:24.005+0000 7f8e78ff9700 1 -- 172.21.15.61:0/761797240 <== mgr.14116 v2:172.21.15.61:6800/19001851 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f8e7c0fc9d0 con 0x7f8e6802fdb0 2023-12-25T21:06:24.006 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: value unchanged 2023-12-25T21:06:24.008 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:24.007+0000 7f8e82902700 1 -- 172.21.15.61:0/761797240 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f8e6802fdb0 msgr2=0x7f8e68032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:24.008 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:24.007+0000 7f8e82902700 1 --2- 172.21.15.61:0/761797240 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f8e6802fdb0 0x7f8e68032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f8e700096a0 tx=0x7f8e70006b40).stop 2023-12-25T21:06:24.009 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:24.007+0000 7f8e82902700 1 -- 172.21.15.61:0/761797240 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e7c0fab80 msgr2=0x7f8e7c10d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:24.009 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:24.007+0000 7f8e82902700 1 --2- 172.21.15.61:0/761797240 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e7c0fab80 0x7f8e7c10d720 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f8e6c00e040 tx=0x7f8e6c006ee0).stop 2023-12-25T21:06:24.009 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:24.007+0000 7f8e82902700 1 -- 172.21.15.61:0/761797240 shutdown_connections 2023-12-25T21:06:24.009 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:24.007+0000 7f8e82902700 1 --2- 172.21.15.61:0/761797240 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f8e6802fdb0 0x7f8e68032270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:24.009 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:24.007+0000 7f8e82902700 1 --2- 172.21.15.61:0/761797240 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e7c0fab80 0x7f8e7c10d720 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:24.009 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:24.007+0000 7f8e82902700 1 -- 172.21.15.61:0/761797240 >> 172.21.15.61:0/761797240 conn(0x7f8e7c0f63c0 msgr2=0x7f8e7c0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:24.009 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:24.007+0000 7f8e82902700 1 -- 172.21.15.61:0/761797240 shutdown_connections 2023-12-25T21:06:24.009 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:24.007+0000 7f8e82902700 1 -- 172.21.15.61:0/761797240 wait complete. 2023-12-25T21:06:24.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:24 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:24.006510+0000 mgr.smithi061.jxxeke (mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T21:06:25.972 INFO:teuthology.orchestra.run.smithi061.stderr:Generating ssh key... 2023-12-25T21:06:25.972 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2023-12-25T21:06:27.264 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.165+0000 7fe069e0c700 1 Processor -- start 2023-12-25T21:06:27.264 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.165+0000 7fe069e0c700 1 -- start start 2023-12-25T21:06:27.264 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.165+0000 7fe069e0c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0640fab80 0x7fe0640fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:27.264 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.165+0000 7fe069e0c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0640fb490 con 0x7fe0640fab80 2023-12-25T21:06:27.264 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.165+0000 7fe0637fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0640fab80 0x7fe0640fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:27.265 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.165+0000 7fe0637fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0640fab80 0x7fe0640fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42926/0 (socket says 172.21.15.61:42926) 2023-12-25T21:06:27.265 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.165+0000 7fe0637fe700 1 -- 172.21.15.61:0/3638513266 learned_addr learned my addr 172.21.15.61:0/3638513266 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:27.265 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.166+0000 7fe0637fe700 1 -- 172.21.15.61:0/3638513266 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0640fb5d0 con 0x7fe0640fab80 2023-12-25T21:06:27.265 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.166+0000 7fe0637fe700 1 --2- 172.21.15.61:0/3638513266 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0640fab80 0x7fe0640fafa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fe05400b5f0 tx=0x7fe054006ee0).ready entity=mon.0 client_cookie=748be12944a9124a server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:27.265 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.166+0000 7fe0627fc700 1 -- 172.21.15.61:0/3638513266 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe05400d600 con 0x7fe0640fab80 2023-12-25T21:06:27.265 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.166+0000 7fe0627fc700 1 -- 172.21.15.61:0/3638513266 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe05400dbd0 con 0x7fe0640fab80 2023-12-25T21:06:27.265 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.166+0000 7fe0627fc700 1 -- 172.21.15.61:0/3638513266 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe054011c00 con 0x7fe0640fab80 2023-12-25T21:06:27.265 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.166+0000 7fe069e0c700 1 -- 172.21.15.61:0/3638513266 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0640fab80 msgr2=0x7fe0640fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:27.265 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.166+0000 7fe069e0c700 1 --2- 172.21.15.61:0/3638513266 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0640fab80 0x7fe0640fafa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fe05400b5f0 tx=0x7fe054006ee0).stop 2023-12-25T21:06:27.265 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.166+0000 7fe069e0c700 1 -- 172.21.15.61:0/3638513266 shutdown_connections 2023-12-25T21:06:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.166+0000 7fe069e0c700 1 --2- 172.21.15.61:0/3638513266 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0640fab80 0x7fe0640fafa0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.166+0000 7fe069e0c700 1 -- 172.21.15.61:0/3638513266 >> 172.21.15.61:0/3638513266 conn(0x7fe0640f63c0 msgr2=0x7fe0640f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.167+0000 7fe069e0c700 1 -- 172.21.15.61:0/3638513266 shutdown_connections 2023-12-25T21:06:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.167+0000 7fe069e0c700 1 -- 172.21.15.61:0/3638513266 wait complete. 2023-12-25T21:06:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.167+0000 7fe069e0c700 1 Processor -- start 2023-12-25T21:06:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.167+0000 7fe069e0c700 1 -- start start 2023-12-25T21:06:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.167+0000 7fe069e0c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0640fab80 0x7fe064100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.167+0000 7fe069e0c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0640ff290 con 0x7fe0640fab80 2023-12-25T21:06:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.168+0000 7fe0637fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0640fab80 0x7fe064100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.168+0000 7fe0637fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0640fab80 0x7fe064100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42942/0 (socket says 172.21.15.61:42942) 2023-12-25T21:06:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.168+0000 7fe0637fe700 1 -- 172.21.15.61:0/1756095736 learned_addr learned my addr 172.21.15.61:0/1756095736 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.168+0000 7fe0637fe700 1 -- 172.21.15.61:0/1756095736 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05400b200 con 0x7fe0640fab80 2023-12-25T21:06:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.168+0000 7fe0637fe700 1 --2- 172.21.15.61:0/1756095736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0640fab80 0x7fe064100e10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fe05400b1d0 tx=0x7fe054006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:27.267 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.168+0000 7fe04bfff700 1 -- 172.21.15.61:0/1756095736 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe05401b410 con 0x7fe0640fab80 2023-12-25T21:06:27.267 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.168+0000 7fe04bfff700 1 -- 172.21.15.61:0/1756095736 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe05401b9e0 con 0x7fe0640fab80 2023-12-25T21:06:27.267 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.168+0000 7fe04bfff700 1 -- 172.21.15.61:0/1756095736 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe05401a840 con 0x7fe0640fab80 2023-12-25T21:06:27.267 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.168+0000 7fe069e0c700 1 -- 172.21.15.61:0/1756095736 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0640ff430 con 0x7fe0640fab80 2023-12-25T21:06:27.267 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.168+0000 7fe069e0c700 1 -- 172.21.15.61:0/1756095736 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0640ff840 con 0x7fe0640fab80 2023-12-25T21:06:27.267 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.169+0000 7fe04bfff700 1 -- 172.21.15.61:0/1756095736 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fe05400f070 con 0x7fe0640fab80 2023-12-25T21:06:27.267 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.169+0000 7fe04bfff700 1 --2- 172.21.15.61:0/1756095736 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7fe05002fdb0 0x7fe050032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:27.267 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.169+0000 7fe04bfff700 1 -- 172.21.15.61:0/1756095736 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fe054041190 con 0x7fe0640fab80 2023-12-25T21:06:27.267 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.169+0000 7fe069e0c700 1 -- 172.21.15.61:0/1756095736 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe064045ad0 con 0x7fe0640fab80 2023-12-25T21:06:27.267 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.169+0000 7fe062ffd700 1 --2- 172.21.15.61:0/1756095736 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7fe05002fdb0 0x7fe050032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:27.268 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.170+0000 7fe062ffd700 1 --2- 172.21.15.61:0/1756095736 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7fe05002fdb0 0x7fe050032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe0580096a0 tx=0x7fe058006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:27.268 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.174+0000 7fe04bfff700 1 -- 172.21.15.61:0/1756095736 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe054016020 con 0x7fe0640fab80 2023-12-25T21:06:27.338 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.336+0000 7fe069e0c700 1 -- 172.21.15.61:0/1756095736 --> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0641049d0 con 0x7fe05002fdb0 2023-12-25T21:06:27.535 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.533+0000 7fe04bfff700 1 -- 172.21.15.61:0/1756095736 <== mgr.14116 v2:172.21.15.61:6800/19001851 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe0641049d0 con 0x7fe05002fdb0 2023-12-25T21:06:27.537 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.535+0000 7fe069e0c700 1 -- 172.21.15.61:0/1756095736 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7fe05002fdb0 msgr2=0x7fe050032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:27.537 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.535+0000 7fe069e0c700 1 --2- 172.21.15.61:0/1756095736 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7fe05002fdb0 0x7fe050032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe0580096a0 tx=0x7fe058006b40).stop 2023-12-25T21:06:27.538 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.535+0000 7fe069e0c700 1 -- 172.21.15.61:0/1756095736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0640fab80 msgr2=0x7fe064100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:27.538 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.535+0000 7fe069e0c700 1 --2- 172.21.15.61:0/1756095736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0640fab80 0x7fe064100e10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fe05400b1d0 tx=0x7fe054006ee0).stop 2023-12-25T21:06:27.538 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.535+0000 7fe069e0c700 1 -- 172.21.15.61:0/1756095736 shutdown_connections 2023-12-25T21:06:27.538 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.535+0000 7fe069e0c700 1 --2- 172.21.15.61:0/1756095736 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7fe05002fdb0 0x7fe050032270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:27.538 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.535+0000 7fe069e0c700 1 --2- 172.21.15.61:0/1756095736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0640fab80 0x7fe064100e10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:27.541 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.535+0000 7fe069e0c700 1 -- 172.21.15.61:0/1756095736 >> 172.21.15.61:0/1756095736 conn(0x7fe0640f63c0 msgr2=0x7fe064070090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:27.541 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.535+0000 7fe069e0c700 1 -- 172.21.15.61:0/1756095736 shutdown_connections 2023-12-25T21:06:27.541 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:27.535+0000 7fe069e0c700 1 -- 172.21.15.61:0/1756095736 wait complete. 2023-12-25T21:06:28.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:28 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cephadm 2023-12-25T21:06:27.339415+0000 mgr.smithi061.jxxeke (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2023-12-25T21:06:28.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:28 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:27.533086+0000 mon.smithi061 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:28.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:28 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:27.534785+0000 mon.smithi061 (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:29.306 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2023-12-25T21:06:29.656 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.653+0000 7fa6b3bd4700 1 Processor -- start 2023-12-25T21:06:29.656 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.653+0000 7fa6b3bd4700 1 -- start start 2023-12-25T21:06:29.656 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.653+0000 7fa6b3bd4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ac070430 0x7fa6ac0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:29.656 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.653+0000 7fa6b3bd4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac0fedd0 con 0x7fa6ac070430 2023-12-25T21:06:29.657 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.654+0000 7fa6b1970700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ac070430 0x7fa6ac0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:29.657 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.654+0000 7fa6b1970700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ac070430 0x7fa6ac0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59166/0 (socket says 172.21.15.61:59166) 2023-12-25T21:06:29.657 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.654+0000 7fa6b1970700 1 -- 172.21.15.61:0/278421122 learned_addr learned my addr 172.21.15.61:0/278421122 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:29.657 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.654+0000 7fa6b1970700 1 -- 172.21.15.61:0/278421122 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac0fef10 con 0x7fa6ac070430 2023-12-25T21:06:29.657 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.654+0000 7fa6b1970700 1 --2- 172.21.15.61:0/278421122 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ac070430 0x7fa6ac0fe8e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fa6a800b5f0 tx=0x7fa6a8006ee0).ready entity=mon.0 client_cookie=d3ecc59e6b94a530 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:29.657 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.654+0000 7fa6b096e700 1 -- 172.21.15.61:0/278421122 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6a800d600 con 0x7fa6ac070430 2023-12-25T21:06:29.657 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.654+0000 7fa6b096e700 1 -- 172.21.15.61:0/278421122 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa6a800dbd0 con 0x7fa6ac070430 2023-12-25T21:06:29.657 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.654+0000 7fa6b096e700 1 -- 172.21.15.61:0/278421122 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6a8011c00 con 0x7fa6ac070430 2023-12-25T21:06:29.657 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.655+0000 7fa6b3bd4700 1 -- 172.21.15.61:0/278421122 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ac070430 msgr2=0x7fa6ac0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:29.659 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.655+0000 7fa6b3bd4700 1 --2- 172.21.15.61:0/278421122 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ac070430 0x7fa6ac0fe8e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fa6a800b5f0 tx=0x7fa6a8006ee0).stop 2023-12-25T21:06:29.659 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.655+0000 7fa6b3bd4700 1 -- 172.21.15.61:0/278421122 shutdown_connections 2023-12-25T21:06:29.659 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.655+0000 7fa6b3bd4700 1 --2- 172.21.15.61:0/278421122 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ac070430 0x7fa6ac0fe8e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:29.659 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.655+0000 7fa6b3bd4700 1 -- 172.21.15.61:0/278421122 >> 172.21.15.61:0/278421122 conn(0x7fa6ac0f63c0 msgr2=0x7fa6ac0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:29.659 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.655+0000 7fa6b3bd4700 1 -- 172.21.15.61:0/278421122 shutdown_connections 2023-12-25T21:06:29.659 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.655+0000 7fa6b3bd4700 1 -- 172.21.15.61:0/278421122 wait complete. 2023-12-25T21:06:29.659 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.656+0000 7fa6b3bd4700 1 Processor -- start 2023-12-25T21:06:29.660 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.656+0000 7fa6b3bd4700 1 -- start start 2023-12-25T21:06:29.660 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.656+0000 7fa6b3bd4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ac070430 0x7fa6ac10aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:29.660 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.656+0000 7fa6b3bd4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac10af30 con 0x7fa6ac070430 2023-12-25T21:06:29.660 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.656+0000 7fa6b1970700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ac070430 0x7fa6ac10aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:29.660 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.656+0000 7fa6b1970700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ac070430 0x7fa6ac10aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59182/0 (socket says 172.21.15.61:59182) 2023-12-25T21:06:29.660 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.656+0000 7fa6b1970700 1 -- 172.21.15.61:0/131169862 learned_addr learned my addr 172.21.15.61:0/131169862 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:29.660 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.656+0000 7fa6b1970700 1 -- 172.21.15.61:0/131169862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a800b200 con 0x7fa6ac070430 2023-12-25T21:06:29.660 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.656+0000 7fa6b1970700 1 --2- 172.21.15.61:0/131169862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ac070430 0x7fa6ac10aa40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa6a800e040 tx=0x7fa6a8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:29.661 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.657+0000 7fa69a7fc700 1 -- 172.21.15.61:0/131169862 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6a800d9f0 con 0x7fa6ac070430 2023-12-25T21:06:29.661 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.657+0000 7fa69a7fc700 1 -- 172.21.15.61:0/131169862 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa6a801a9f0 con 0x7fa6ac070430 2023-12-25T21:06:29.661 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.657+0000 7fa69a7fc700 1 -- 172.21.15.61:0/131169862 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6a801b8c0 con 0x7fa6ac070430 2023-12-25T21:06:29.661 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.657+0000 7fa6b3bd4700 1 -- 172.21.15.61:0/131169862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ac10b130 con 0x7fa6ac070430 2023-12-25T21:06:29.661 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.657+0000 7fa6b3bd4700 1 -- 172.21.15.61:0/131169862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ac10b540 con 0x7fa6ac070430 2023-12-25T21:06:29.661 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.657+0000 7fa69a7fc700 1 -- 172.21.15.61:0/131169862 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa6a800f070 con 0x7fa6ac070430 2023-12-25T21:06:29.661 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.657+0000 7fa69a7fc700 1 --2- 172.21.15.61:0/131169862 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7fa69402fdb0 0x7fa694032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:29.661 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.658+0000 7fa69a7fc700 1 -- 172.21.15.61:0/131169862 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fa6a8041260 con 0x7fa6ac070430 2023-12-25T21:06:29.661 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.658+0000 7fa6b116f700 1 --2- 172.21.15.61:0/131169862 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7fa69402fdb0 0x7fa694032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:29.662 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.658+0000 7fa6b116f700 1 --2- 172.21.15.61:0/131169862 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7fa69402fdb0 0x7fa694032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa69c0096a0 tx=0x7fa69c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:29.662 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.658+0000 7fa6b3bd4700 1 -- 172.21.15.61:0/131169862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ac059100 con 0x7fa6ac070430 2023-12-25T21:06:29.664 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.662+0000 7fa69a7fc700 1 -- 172.21.15.61:0/131169862 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6a801a580 con 0x7fa6ac070430 2023-12-25T21:06:29.826 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.824+0000 7fa6b3bd4700 1 -- 172.21.15.61:0/131169862 --> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6ac104890 con 0x7fa69402fdb0 2023-12-25T21:06:29.826 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.824+0000 7fa69a7fc700 1 -- 172.21.15.61:0/131169862 <== mgr.14116 v2:172.21.15.61:6800/19001851 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fa6ac104890 con 0x7fa69402fdb0 2023-12-25T21:06:29.826 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDF1CLZFIijCNuXLHAYeAVzgtdpgjqt+gv9o3wbHv7twCQ2/+pUUZS1i/ATYzdlLQRQ+vmGLLi31BwaDPxIPJVqOKm2eZkiQmwKkNZ/7UmnBXoFgB/do8+StfyNQXxjLYiupT7nrwOp8ZpEDAQAcz4vAK8SUHvbkLnWn7Lk1nwkU8CG1GRDwYVWPGCWkGaSXc2oto0Xf1nZbsx2JKqJ5copwy3BKzOGY4m+0OuVL2phf0ST709MzY3Q3mtsnW2T0SjwGINQltMI/4ukXFXhIwxQSO3+5DCaC19C8LJCcqpSOz8/Zr0x9P43a6FgzWbWtpwsZNHbxcqktHC9u8UWLDi9Hp6E2r3UzTihJ1FVUWTvwMf85ol/EtLSssQJKzJ0E5rZ7D8/tpFKRKhb76I5VXR59NJhMqwEzUMNaU6ewQUnbz0UjvwK6ysjpI0dwpX+VY4KeaWI854H3Uwfzt+WvHduM3K0Q0yr2tbZscLREGw92M8i9oa7t5mP9Ntc7tgHI+8= ceph-5532d712-a369-11ee-95a5-87774f69a715 2023-12-25T21:06:29.829 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.826+0000 7fa6b3bd4700 1 -- 172.21.15.61:0/131169862 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7fa69402fdb0 msgr2=0x7fa694032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:29.829 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.826+0000 7fa6b3bd4700 1 --2- 172.21.15.61:0/131169862 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7fa69402fdb0 0x7fa694032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa69c0096a0 tx=0x7fa69c006b40).stop 2023-12-25T21:06:29.829 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.826+0000 7fa6b3bd4700 1 -- 172.21.15.61:0/131169862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ac070430 msgr2=0x7fa6ac10aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:29.829 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.826+0000 7fa6b3bd4700 1 --2- 172.21.15.61:0/131169862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ac070430 0x7fa6ac10aa40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa6a800e040 tx=0x7fa6a8006ee0).stop 2023-12-25T21:06:29.829 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.826+0000 7fa6b3bd4700 1 -- 172.21.15.61:0/131169862 shutdown_connections 2023-12-25T21:06:29.829 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.826+0000 7fa6b3bd4700 1 --2- 172.21.15.61:0/131169862 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7fa69402fdb0 0x7fa694032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:29.829 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.826+0000 7fa6b3bd4700 1 --2- 172.21.15.61:0/131169862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6ac070430 0x7fa6ac10aa40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:29.829 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.826+0000 7fa6b3bd4700 1 -- 172.21.15.61:0/131169862 >> 172.21.15.61:0/131169862 conn(0x7fa6ac0f63c0 msgr2=0x7fa6ac0f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:29.830 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.827+0000 7fa6b3bd4700 1 -- 172.21.15.61:0/131169862 shutdown_connections 2023-12-25T21:06:29.830 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:29.827+0000 7fa6b3bd4700 1 -- 172.21.15.61:0/131169862 wait complete. 2023-12-25T21:06:30.171 INFO:teuthology.orchestra.run.smithi061.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-12-25T21:06:30.171 INFO:teuthology.orchestra.run.smithi061.stderr:Adding key to root@localhost authorized_keys... 2023-12-25T21:06:30.171 INFO:teuthology.orchestra.run.smithi061.stderr:Adding host smithi061... 2023-12-25T21:06:30.171 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi061 172.21.15.61 2023-12-25T21:06:30.523 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.520+0000 7f2312446700 1 Processor -- start 2023-12-25T21:06:30.523 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.520+0000 7f2312446700 1 -- start start 2023-12-25T21:06:30.523 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.520+0000 7f2312446700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f230c0fab80 0x7f230c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:30.524 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.520+0000 7f2312446700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230c0fb490 con 0x7f230c0fab80 2023-12-25T21:06:30.524 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.521+0000 7f230bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f230c0fab80 0x7f230c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:30.524 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.521+0000 7f230bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f230c0fab80 0x7f230c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59198/0 (socket says 172.21.15.61:59198) 2023-12-25T21:06:30.524 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.521+0000 7f230bfff700 1 -- 172.21.15.61:0/3447288980 learned_addr learned my addr 172.21.15.61:0/3447288980 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:30.524 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.521+0000 7f230bfff700 1 -- 172.21.15.61:0/3447288980 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230c0fb5d0 con 0x7f230c0fab80 2023-12-25T21:06:30.524 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.521+0000 7f230bfff700 1 --2- 172.21.15.61:0/3447288980 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f230c0fab80 0x7f230c0fafa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f230000b870 tx=0x7f2300006f70).ready entity=mon.0 client_cookie=2c654d4babd6ab29 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:30.524 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.521+0000 7f230affd700 1 -- 172.21.15.61:0/3447288980 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f230000da20 con 0x7f230c0fab80 2023-12-25T21:06:30.524 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.521+0000 7f230affd700 1 -- 172.21.15.61:0/3447288980 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2300010470 con 0x7f230c0fab80 2023-12-25T21:06:30.524 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.521+0000 7f230affd700 1 -- 172.21.15.61:0/3447288980 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f230000f5b0 con 0x7f230c0fab80 2023-12-25T21:06:30.527 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.522+0000 7f2312446700 1 -- 172.21.15.61:0/3447288980 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f230c0fab80 msgr2=0x7f230c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:30.527 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.522+0000 7f2312446700 1 --2- 172.21.15.61:0/3447288980 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f230c0fab80 0x7f230c0fafa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f230000b870 tx=0x7f2300006f70).stop 2023-12-25T21:06:30.527 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.522+0000 7f2312446700 1 -- 172.21.15.61:0/3447288980 shutdown_connections 2023-12-25T21:06:30.527 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.522+0000 7f2312446700 1 --2- 172.21.15.61:0/3447288980 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f230c0fab80 0x7f230c0fafa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:30.527 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.522+0000 7f2312446700 1 -- 172.21.15.61:0/3447288980 >> 172.21.15.61:0/3447288980 conn(0x7f230c0f63c0 msgr2=0x7f230c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:30.527 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.522+0000 7f2312446700 1 -- 172.21.15.61:0/3447288980 shutdown_connections 2023-12-25T21:06:30.527 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.522+0000 7f2312446700 1 -- 172.21.15.61:0/3447288980 wait complete. 2023-12-25T21:06:30.527 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.523+0000 7f2312446700 1 Processor -- start 2023-12-25T21:06:30.527 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.523+0000 7f2312446700 1 -- start start 2023-12-25T21:06:30.528 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.523+0000 7f2312446700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f230c0fab80 0x7f230c108780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:30.528 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.523+0000 7f2312446700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230c108c70 con 0x7f230c0fab80 2023-12-25T21:06:30.528 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.523+0000 7f230bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f230c0fab80 0x7f230c108780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:30.528 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.523+0000 7f230bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f230c0fab80 0x7f230c108780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59208/0 (socket says 172.21.15.61:59208) 2023-12-25T21:06:30.528 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.523+0000 7f230bfff700 1 -- 172.21.15.61:0/1977997398 learned_addr learned my addr 172.21.15.61:0/1977997398 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:30.528 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.524+0000 7f230bfff700 1 -- 172.21.15.61:0/1977997398 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230000b480 con 0x7f230c0fab80 2023-12-25T21:06:30.528 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.524+0000 7f230bfff700 1 --2- 172.21.15.61:0/1977997398 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f230c0fab80 0x7f230c108780 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f2300006a40 tx=0x7f2300006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:30.528 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.524+0000 7f2308ff9700 1 -- 172.21.15.61:0/1977997398 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f230000f3d0 con 0x7f230c0fab80 2023-12-25T21:06:30.528 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.524+0000 7f2308ff9700 1 -- 172.21.15.61:0/1977997398 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f230000fef0 con 0x7f230c0fab80 2023-12-25T21:06:30.528 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.524+0000 7f2312446700 1 -- 172.21.15.61:0/1977997398 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f230c108e70 con 0x7f230c0fab80 2023-12-25T21:06:30.528 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.524+0000 7f2308ff9700 1 -- 172.21.15.61:0/1977997398 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2300019d40 con 0x7f230c0fab80 2023-12-25T21:06:30.528 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.524+0000 7f2312446700 1 -- 172.21.15.61:0/1977997398 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f230c109280 con 0x7f230c0fab80 2023-12-25T21:06:30.528 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.525+0000 7f2308ff9700 1 -- 172.21.15.61:0/1977997398 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f230002b060 con 0x7f230c0fab80 2023-12-25T21:06:30.528 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.525+0000 7f2308ff9700 1 --2- 172.21.15.61:0/1977997398 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f22f802fd50 0x7f22f8032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:30.529 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.525+0000 7f2308ff9700 1 -- 172.21.15.61:0/1977997398 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f23000440c0 con 0x7f230c0fab80 2023-12-25T21:06:30.529 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.525+0000 7f230b7fe700 1 --2- 172.21.15.61:0/1977997398 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f22f802fd50 0x7f22f8032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:30.529 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.525+0000 7f230b7fe700 1 --2- 172.21.15.61:0/1977997398 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f22f802fd50 0x7f22f8032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f22fc0096a0 tx=0x7f22fc006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:30.529 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.525+0000 7f2312446700 1 -- 172.21.15.61:0/1977997398 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f230c059100 con 0x7f230c0fab80 2023-12-25T21:06:30.531 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.530+0000 7f2308ff9700 1 -- 172.21.15.61:0/1977997398 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2300014020 con 0x7f230c0fab80 2023-12-25T21:06:30.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:30 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:29.826596+0000 mgr.smithi061.jxxeke (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T21:06:30.712 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:30.710+0000 7f2312446700 1 -- 172.21.15.61:0/1977997398 --> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi061", "addr": "172.21.15.61", "target": ["mon-mgr", ""]}) v1 -- 0x7f230c102640 con 0x7f22f802fd50 2023-12-25T21:06:31.335 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:31.333+0000 7f2308ff9700 1 -- 172.21.15.61:0/1977997398 <== mgr.14116 v2:172.21.15.61:6800/19001851 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f230c102640 con 0x7f22f802fd50 2023-12-25T21:06:31.335 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: Added host 'smithi061' 2023-12-25T21:06:31.336 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:31.335+0000 7f2312446700 1 -- 172.21.15.61:0/1977997398 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f22f802fd50 msgr2=0x7f22f8032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:31.336 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:31.335+0000 7f2312446700 1 --2- 172.21.15.61:0/1977997398 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f22f802fd50 0x7f22f8032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f22fc0096a0 tx=0x7f22fc006b40).stop 2023-12-25T21:06:31.336 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:31.335+0000 7f2312446700 1 -- 172.21.15.61:0/1977997398 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f230c0fab80 msgr2=0x7f230c108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:31.336 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:31.335+0000 7f2312446700 1 --2- 172.21.15.61:0/1977997398 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f230c0fab80 0x7f230c108780 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f2300006a40 tx=0x7f2300006f70).stop 2023-12-25T21:06:31.337 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:31.336+0000 7f2312446700 1 -- 172.21.15.61:0/1977997398 shutdown_connections 2023-12-25T21:06:31.337 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:31.336+0000 7f2312446700 1 --2- 172.21.15.61:0/1977997398 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f22f802fd50 0x7f22f8032210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:31.338 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:31.336+0000 7f2312446700 1 --2- 172.21.15.61:0/1977997398 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f230c0fab80 0x7f230c108780 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:31.338 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:31.336+0000 7f2312446700 1 -- 172.21.15.61:0/1977997398 >> 172.21.15.61:0/1977997398 conn(0x7f230c0f63c0 msgr2=0x7f230c0ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:31.338 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:31.336+0000 7f2312446700 1 -- 172.21.15.61:0/1977997398 shutdown_connections 2023-12-25T21:06:31.338 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:31.336+0000 7f2312446700 1 -- 172.21.15.61:0/1977997398 wait complete. 2023-12-25T21:06:31.662 INFO:teuthology.orchestra.run.smithi061.stderr:Deploying mon service with default placement... 2023-12-25T21:06:31.662 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2023-12-25T21:06:32.010 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.006+0000 7f87b5e62700 1 Processor -- start 2023-12-25T21:06:32.010 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.006+0000 7f87b5e62700 1 -- start start 2023-12-25T21:06:32.010 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.006+0000 7f87b5e62700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b00fab80 0x7f87b00fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:32.010 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.006+0000 7f87b5e62700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b00fb490 con 0x7f87b00fab80 2023-12-25T21:06:32.010 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.007+0000 7f87af7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b00fab80 0x7f87b00fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:32.010 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.007+0000 7f87af7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b00fab80 0x7f87b00fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59224/0 (socket says 172.21.15.61:59224) 2023-12-25T21:06:32.010 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.007+0000 7f87af7fe700 1 -- 172.21.15.61:0/1086193023 learned_addr learned my addr 172.21.15.61:0/1086193023 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.007+0000 7f87af7fe700 1 -- 172.21.15.61:0/1086193023 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b00fb5d0 con 0x7f87b00fab80 2023-12-25T21:06:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.007+0000 7f87af7fe700 1 --2- 172.21.15.61:0/1086193023 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b00fab80 0x7f87b00fafa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f87a000b5f0 tx=0x7f87a0006ee0).ready entity=mon.0 client_cookie=64253160fb069802 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.007+0000 7f87ae7fc700 1 -- 172.21.15.61:0/1086193023 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87a000d600 con 0x7f87b00fab80 2023-12-25T21:06:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.008+0000 7f87ae7fc700 1 -- 172.21.15.61:0/1086193023 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f87a000dbd0 con 0x7f87b00fab80 2023-12-25T21:06:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.008+0000 7f87ae7fc700 1 -- 172.21.15.61:0/1086193023 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87a0011c00 con 0x7f87b00fab80 2023-12-25T21:06:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.008+0000 7f87b5e62700 1 -- 172.21.15.61:0/1086193023 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b00fab80 msgr2=0x7f87b00fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.008+0000 7f87b5e62700 1 --2- 172.21.15.61:0/1086193023 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b00fab80 0x7f87b00fafa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f87a000b5f0 tx=0x7f87a0006ee0).stop 2023-12-25T21:06:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.008+0000 7f87b5e62700 1 -- 172.21.15.61:0/1086193023 shutdown_connections 2023-12-25T21:06:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.008+0000 7f87b5e62700 1 --2- 172.21.15.61:0/1086193023 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b00fab80 0x7f87b00fafa0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.008+0000 7f87b5e62700 1 -- 172.21.15.61:0/1086193023 >> 172.21.15.61:0/1086193023 conn(0x7f87b00f63c0 msgr2=0x7f87b00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.008+0000 7f87b5e62700 1 -- 172.21.15.61:0/1086193023 shutdown_connections 2023-12-25T21:06:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.008+0000 7f87b5e62700 1 -- 172.21.15.61:0/1086193023 wait complete. 2023-12-25T21:06:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.009+0000 7f87b5e62700 1 Processor -- start 2023-12-25T21:06:32.012 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.009+0000 7f87b5e62700 1 -- start start 2023-12-25T21:06:32.012 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.009+0000 7f87b5e62700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b00fab80 0x7f87b0100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:32.012 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.009+0000 7f87b5e62700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b00ff290 con 0x7f87b00fab80 2023-12-25T21:06:32.013 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.009+0000 7f87af7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b00fab80 0x7f87b0100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:32.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.009+0000 7f87af7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b00fab80 0x7f87b0100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59228/0 (socket says 172.21.15.61:59228) 2023-12-25T21:06:32.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.009+0000 7f87af7fe700 1 -- 172.21.15.61:0/1237552315 learned_addr learned my addr 172.21.15.61:0/1237552315 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:32.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.009+0000 7f87af7fe700 1 -- 172.21.15.61:0/1237552315 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a000b200 con 0x7f87b00fab80 2023-12-25T21:06:32.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.010+0000 7f87af7fe700 1 --2- 172.21.15.61:0/1237552315 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b00fab80 0x7f87b0100e10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f87a000b1d0 tx=0x7f87a0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:32.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.010+0000 7f87a7fff700 1 -- 172.21.15.61:0/1237552315 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87a001b410 con 0x7f87b00fab80 2023-12-25T21:06:32.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.010+0000 7f87a7fff700 1 -- 172.21.15.61:0/1237552315 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f87a001b9e0 con 0x7f87b00fab80 2023-12-25T21:06:32.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.010+0000 7f87b5e62700 1 -- 172.21.15.61:0/1237552315 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b00ff430 con 0x7f87b00fab80 2023-12-25T21:06:32.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.010+0000 7f87a7fff700 1 -- 172.21.15.61:0/1237552315 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87a001a840 con 0x7f87b00fab80 2023-12-25T21:06:32.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.010+0000 7f87b5e62700 1 -- 172.21.15.61:0/1237552315 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b00ff840 con 0x7f87b00fab80 2023-12-25T21:06:32.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.011+0000 7f87a7fff700 1 -- 172.21.15.61:0/1237552315 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f87a000f070 con 0x7f87b00fab80 2023-12-25T21:06:32.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.011+0000 7f87a7fff700 1 --2- 172.21.15.61:0/1237552315 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f879002fd50 0x7f8790032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:32.014 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.011+0000 7f87a7fff700 1 -- 172.21.15.61:0/1237552315 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f87a00412d0 con 0x7f87b00fab80 2023-12-25T21:06:32.015 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.011+0000 7f87aeffd700 1 --2- 172.21.15.61:0/1237552315 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f879002fd50 0x7f8790032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:32.015 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.011+0000 7f87b5e62700 1 -- 172.21.15.61:0/1237552315 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b0059100 con 0x7f87b00fab80 2023-12-25T21:06:32.015 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.011+0000 7f87aeffd700 1 --2- 172.21.15.61:0/1237552315 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f879002fd50 0x7f8790032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f87980096a0 tx=0x7f8798006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:32.016 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.015+0000 7f87a7fff700 1 -- 172.21.15.61:0/1237552315 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87a0025460 con 0x7f87b00fab80 2023-12-25T21:06:32.201 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.200+0000 7f87b5e62700 1 -- 172.21.15.61:0/1237552315 --> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f87b0104a00 con 0x7f879002fd50 2023-12-25T21:06:32.204 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.204+0000 7f87a7fff700 1 -- 172.21.15.61:0/1237552315 <== mgr.14116 v2:172.21.15.61:6800/19001851 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f87b0104a00 con 0x7f879002fd50 2023-12-25T21:06:32.205 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: Scheduled mon update... 2023-12-25T21:06:32.207 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.205+0000 7f87b5e62700 1 -- 172.21.15.61:0/1237552315 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f879002fd50 msgr2=0x7f8790032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:32.207 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.206+0000 7f87b5e62700 1 --2- 172.21.15.61:0/1237552315 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f879002fd50 0x7f8790032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f87980096a0 tx=0x7f8798006b40).stop 2023-12-25T21:06:32.207 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.206+0000 7f87b5e62700 1 -- 172.21.15.61:0/1237552315 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b00fab80 msgr2=0x7f87b0100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:32.207 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.206+0000 7f87b5e62700 1 --2- 172.21.15.61:0/1237552315 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b00fab80 0x7f87b0100e10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f87a000b1d0 tx=0x7f87a0006ee0).stop 2023-12-25T21:06:32.207 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.206+0000 7f87b5e62700 1 -- 172.21.15.61:0/1237552315 shutdown_connections 2023-12-25T21:06:32.208 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.206+0000 7f87b5e62700 1 --2- 172.21.15.61:0/1237552315 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f879002fd50 0x7f8790032210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:32.208 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.206+0000 7f87b5e62700 1 --2- 172.21.15.61:0/1237552315 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b00fab80 0x7f87b0100e10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:32.208 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.206+0000 7f87b5e62700 1 -- 172.21.15.61:0/1237552315 >> 172.21.15.61:0/1237552315 conn(0x7f87b00f63c0 msgr2=0x7f87b00f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:32.208 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.206+0000 7f87b5e62700 1 -- 172.21.15.61:0/1237552315 shutdown_connections 2023-12-25T21:06:32.208 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.206+0000 7f87b5e62700 1 -- 172.21.15.61:0/1237552315 wait complete. 2023-12-25T21:06:32.523 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:32 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:30.713366+0000 mgr.smithi061.jxxeke (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi061", "addr": "172.21.15.61", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T21:06:32.523 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:32 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:31.333977+0000 mon.smithi061 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:32.523 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:32 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:31.334581+0000 mon.smithi061 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-25T21:06:32.523 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:32 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:32.204915+0000 mon.smithi061 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:32.545 INFO:teuthology.orchestra.run.smithi061.stderr:Deploying mgr service with default placement... 2023-12-25T21:06:32.545 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2023-12-25T21:06:32.921 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.920+0000 7fd8a214a700 1 Processor -- start 2023-12-25T21:06:32.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.920+0000 7fd8a214a700 1 -- start start 2023-12-25T21:06:32.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.921+0000 7fd8a214a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd89c0703f0 0x7fd89c0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:32.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.921+0000 7fd8a214a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89c0fee10 con 0x7fd89c0703f0 2023-12-25T21:06:32.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.921+0000 7fd89b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd89c0703f0 0x7fd89c0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:32.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.921+0000 7fd89b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd89c0703f0 0x7fd89c0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59238/0 (socket says 172.21.15.61:59238) 2023-12-25T21:06:32.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.921+0000 7fd89b7fe700 1 -- 172.21.15.61:0/83674226 learned_addr learned my addr 172.21.15.61:0/83674226 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:32.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.921+0000 7fd89b7fe700 1 -- 172.21.15.61:0/83674226 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89c0fef50 con 0x7fd89c0703f0 2023-12-25T21:06:32.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.921+0000 7fd89b7fe700 1 --2- 172.21.15.61:0/83674226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd89c0703f0 0x7fd89c0fe920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fd88c00b5f0 tx=0x7fd88c006ee0).ready entity=mon.0 client_cookie=7e7cbe84e58eeb6f server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:32.924 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.921+0000 7fd89a7fc700 1 -- 172.21.15.61:0/83674226 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd88c00d600 con 0x7fd89c0703f0 2023-12-25T21:06:32.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.921+0000 7fd89a7fc700 1 -- 172.21.15.61:0/83674226 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd88c00dbd0 con 0x7fd89c0703f0 2023-12-25T21:06:32.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.921+0000 7fd89a7fc700 1 -- 172.21.15.61:0/83674226 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd88c011c00 con 0x7fd89c0703f0 2023-12-25T21:06:32.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.922+0000 7fd8a214a700 1 -- 172.21.15.61:0/83674226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd89c0703f0 msgr2=0x7fd89c0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:32.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.922+0000 7fd8a214a700 1 --2- 172.21.15.61:0/83674226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd89c0703f0 0x7fd89c0fe920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fd88c00b5f0 tx=0x7fd88c006ee0).stop 2023-12-25T21:06:32.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.922+0000 7fd8a214a700 1 -- 172.21.15.61:0/83674226 shutdown_connections 2023-12-25T21:06:32.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.922+0000 7fd8a214a700 1 --2- 172.21.15.61:0/83674226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd89c0703f0 0x7fd89c0fe920 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:32.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.922+0000 7fd8a214a700 1 -- 172.21.15.61:0/83674226 >> 172.21.15.61:0/83674226 conn(0x7fd89c0f63c0 msgr2=0x7fd89c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:32.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.922+0000 7fd8a214a700 1 -- 172.21.15.61:0/83674226 shutdown_connections 2023-12-25T21:06:32.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.922+0000 7fd8a214a700 1 -- 172.21.15.61:0/83674226 wait complete. 2023-12-25T21:06:32.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.923+0000 7fd8a214a700 1 Processor -- start 2023-12-25T21:06:32.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.923+0000 7fd8a214a700 1 -- start start 2023-12-25T21:06:32.925 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.923+0000 7fd8a214a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd89c0703f0 0x7fd89c0fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:32.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.923+0000 7fd8a214a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89c0fa820 con 0x7fd89c0703f0 2023-12-25T21:06:32.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.923+0000 7fd89b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd89c0703f0 0x7fd89c0fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:32.926 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.923+0000 7fd89b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd89c0703f0 0x7fd89c0fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59250/0 (socket says 172.21.15.61:59250) 2023-12-25T21:06:32.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.923+0000 7fd89b7fe700 1 -- 172.21.15.61:0/2656679817 learned_addr learned my addr 172.21.15.61:0/2656679817 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:32.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.923+0000 7fd89b7fe700 1 -- 172.21.15.61:0/2656679817 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd88c00b200 con 0x7fd89c0703f0 2023-12-25T21:06:32.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.924+0000 7fd89b7fe700 1 --2- 172.21.15.61:0/2656679817 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd89c0703f0 0x7fd89c0fc3a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fd88c00e040 tx=0x7fd88c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:32.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.924+0000 7fd893fff700 1 -- 172.21.15.61:0/2656679817 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd88c00d9f0 con 0x7fd89c0703f0 2023-12-25T21:06:32.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.924+0000 7fd893fff700 1 -- 172.21.15.61:0/2656679817 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd88c01a9f0 con 0x7fd89c0703f0 2023-12-25T21:06:32.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.924+0000 7fd8a214a700 1 -- 172.21.15.61:0/2656679817 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd89c0fa9c0 con 0x7fd89c0703f0 2023-12-25T21:06:32.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.924+0000 7fd893fff700 1 -- 172.21.15.61:0/2656679817 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd88c01b8c0 con 0x7fd89c0703f0 2023-12-25T21:06:32.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.924+0000 7fd8a214a700 1 -- 172.21.15.61:0/2656679817 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd89c0fadd0 con 0x7fd89c0703f0 2023-12-25T21:06:32.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.925+0000 7fd893fff700 1 -- 172.21.15.61:0/2656679817 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fd88c00f070 con 0x7fd89c0703f0 2023-12-25T21:06:32.927 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.925+0000 7fd893fff700 1 --2- 172.21.15.61:0/2656679817 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7fd87c02fd50 0x7fd87c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:32.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.925+0000 7fd8a214a700 1 -- 172.21.15.61:0/2656679817 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd89c045ad0 con 0x7fd89c0703f0 2023-12-25T21:06:32.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.925+0000 7fd893fff700 1 -- 172.21.15.61:0/2656679817 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd88c01ab60 con 0x7fd89c0703f0 2023-12-25T21:06:32.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.925+0000 7fd89affd700 1 --2- 172.21.15.61:0/2656679817 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7fd87c02fd50 0x7fd87c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:32.928 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.925+0000 7fd89affd700 1 --2- 172.21.15.61:0/2656679817 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7fd87c02fd50 0x7fd87c032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fd8840096a0 tx=0x7fd884006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:32.929 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:32.929+0000 7fd893fff700 1 -- 172.21.15.61:0/2656679817 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd88c016070 con 0x7fd89c0703f0 2023-12-25T21:06:33.115 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.114+0000 7fd8a214a700 1 -- 172.21.15.61:0/2656679817 --> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fd89c0fb010 con 0x7fd87c02fd50 2023-12-25T21:06:33.118 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.118+0000 7fd893fff700 1 -- 172.21.15.61:0/2656679817 <== mgr.14116 v2:172.21.15.61:6800/19001851 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd89c0fb010 con 0x7fd87c02fd50 2023-12-25T21:06:33.118 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-12-25T21:06:33.120 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.119+0000 7fd8a214a700 1 -- 172.21.15.61:0/2656679817 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7fd87c02fd50 msgr2=0x7fd87c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:33.121 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.119+0000 7fd8a214a700 1 --2- 172.21.15.61:0/2656679817 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7fd87c02fd50 0x7fd87c032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fd8840096a0 tx=0x7fd884006b40).stop 2023-12-25T21:06:33.121 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.119+0000 7fd8a214a700 1 -- 172.21.15.61:0/2656679817 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd89c0703f0 msgr2=0x7fd89c0fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:33.121 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.119+0000 7fd8a214a700 1 --2- 172.21.15.61:0/2656679817 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd89c0703f0 0x7fd89c0fc3a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fd88c00e040 tx=0x7fd88c006ee0).stop 2023-12-25T21:06:33.121 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.119+0000 7fd8a214a700 1 -- 172.21.15.61:0/2656679817 shutdown_connections 2023-12-25T21:06:33.121 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.119+0000 7fd8a214a700 1 --2- 172.21.15.61:0/2656679817 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7fd87c02fd50 0x7fd87c032210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:33.121 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.119+0000 7fd8a214a700 1 --2- 172.21.15.61:0/2656679817 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd89c0703f0 0x7fd89c0fc3a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:33.121 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.119+0000 7fd8a214a700 1 -- 172.21.15.61:0/2656679817 >> 172.21.15.61:0/2656679817 conn(0x7fd89c0f63c0 msgr2=0x7fd89c0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:33.121 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.119+0000 7fd8a214a700 1 -- 172.21.15.61:0/2656679817 shutdown_connections 2023-12-25T21:06:33.121 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.119+0000 7fd8a214a700 1 -- 172.21.15.61:0/2656679817 wait complete. 2023-12-25T21:06:33.455 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:33 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cephadm 2023-12-25T21:06:31.334309+0000 mgr.smithi061.jxxeke (mgr.14116) 9 : cephadm [INF] Added host smithi061 2023-12-25T21:06:33.455 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:33 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:32.202044+0000 mgr.smithi061.jxxeke (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T21:06:33.455 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:33 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cephadm 2023-12-25T21:06:32.202782+0000 mgr.smithi061.jxxeke (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-12-25T21:06:33.456 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:33 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:33.118616+0000 mon.smithi061 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:33.487 INFO:teuthology.orchestra.run.smithi061.stderr:Deploying crash service with default placement... 2023-12-25T21:06:33.488 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2023-12-25T21:06:33.842 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.840+0000 7f7e36948700 1 Processor -- start 2023-12-25T21:06:33.842 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.840+0000 7f7e36948700 1 -- start start 2023-12-25T21:06:33.843 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.841+0000 7f7e36948700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e300fab80 0x7f7e300fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:33.843 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.841+0000 7f7e36948700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e300fb490 con 0x7f7e300fab80 2023-12-25T21:06:33.843 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.841+0000 7f7e2ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e300fab80 0x7f7e300fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:33.843 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.841+0000 7f7e2ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e300fab80 0x7f7e300fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59254/0 (socket says 172.21.15.61:59254) 2023-12-25T21:06:33.843 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.841+0000 7f7e2ffff700 1 -- 172.21.15.61:0/3562379075 learned_addr learned my addr 172.21.15.61:0/3562379075 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:33.843 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.841+0000 7f7e2ffff700 1 -- 172.21.15.61:0/3562379075 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e300fb5d0 con 0x7f7e300fab80 2023-12-25T21:06:33.843 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.841+0000 7f7e2ffff700 1 --2- 172.21.15.61:0/3562379075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e300fab80 0x7f7e300fafa0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f7e2000b5f0 tx=0x7f7e20006ee0).ready entity=mon.0 client_cookie=b315a23756460f6e server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:33.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.841+0000 7f7e2effd700 1 -- 172.21.15.61:0/3562379075 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e2000d600 con 0x7f7e300fab80 2023-12-25T21:06:33.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.841+0000 7f7e2effd700 1 -- 172.21.15.61:0/3562379075 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7e2000dbd0 con 0x7f7e300fab80 2023-12-25T21:06:33.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.841+0000 7f7e2effd700 1 -- 172.21.15.61:0/3562379075 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e20011c00 con 0x7f7e300fab80 2023-12-25T21:06:33.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.842+0000 7f7e36948700 1 -- 172.21.15.61:0/3562379075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e300fab80 msgr2=0x7f7e300fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:33.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.842+0000 7f7e36948700 1 --2- 172.21.15.61:0/3562379075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e300fab80 0x7f7e300fafa0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f7e2000b5f0 tx=0x7f7e20006ee0).stop 2023-12-25T21:06:33.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.842+0000 7f7e36948700 1 -- 172.21.15.61:0/3562379075 shutdown_connections 2023-12-25T21:06:33.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.842+0000 7f7e36948700 1 --2- 172.21.15.61:0/3562379075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e300fab80 0x7f7e300fafa0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:33.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.842+0000 7f7e36948700 1 -- 172.21.15.61:0/3562379075 >> 172.21.15.61:0/3562379075 conn(0x7f7e300f63c0 msgr2=0x7f7e300f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:33.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.842+0000 7f7e36948700 1 -- 172.21.15.61:0/3562379075 shutdown_connections 2023-12-25T21:06:33.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.842+0000 7f7e36948700 1 -- 172.21.15.61:0/3562379075 wait complete. 2023-12-25T21:06:33.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.842+0000 7f7e36948700 1 Processor -- start 2023-12-25T21:06:33.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.843+0000 7f7e36948700 1 -- start start 2023-12-25T21:06:33.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.843+0000 7f7e36948700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e300fab80 0x7f7e3010d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:33.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.843+0000 7f7e36948700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e30070450 con 0x7f7e300fab80 2023-12-25T21:06:33.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.843+0000 7f7e2ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e300fab80 0x7f7e3010d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:33.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.843+0000 7f7e2ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e300fab80 0x7f7e3010d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59264/0 (socket says 172.21.15.61:59264) 2023-12-25T21:06:33.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.843+0000 7f7e2ffff700 1 -- 172.21.15.61:0/3653678999 learned_addr learned my addr 172.21.15.61:0/3653678999 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:33.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.843+0000 7f7e2ffff700 1 -- 172.21.15.61:0/3653678999 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2000b200 con 0x7f7e300fab80 2023-12-25T21:06:33.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.843+0000 7f7e2ffff700 1 --2- 172.21.15.61:0/3653678999 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e300fab80 0x7f7e3010d740 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f7e2000e040 tx=0x7f7e20006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:33.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.844+0000 7f7e2cff9700 1 -- 172.21.15.61:0/3653678999 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e2001b410 con 0x7f7e300fab80 2023-12-25T21:06:33.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.844+0000 7f7e2cff9700 1 -- 172.21.15.61:0/3653678999 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7e2001b9e0 con 0x7f7e300fab80 2023-12-25T21:06:33.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.844+0000 7f7e36948700 1 -- 172.21.15.61:0/3653678999 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e30070650 con 0x7f7e300fab80 2023-12-25T21:06:33.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.844+0000 7f7e2cff9700 1 -- 172.21.15.61:0/3653678999 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e2001a820 con 0x7f7e300fab80 2023-12-25T21:06:33.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.844+0000 7f7e36948700 1 -- 172.21.15.61:0/3653678999 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e3010ddf0 con 0x7f7e300fab80 2023-12-25T21:06:33.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.844+0000 7f7e2cff9700 1 -- 172.21.15.61:0/3653678999 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f7e2000f070 con 0x7f7e300fab80 2023-12-25T21:06:33.847 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.845+0000 7f7e2cff9700 1 --2- 172.21.15.61:0/3653678999 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f7e1c02fd50 0x7f7e1c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:33.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.845+0000 7f7e2cff9700 1 -- 172.21.15.61:0/3653678999 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f7e20041480 con 0x7f7e300fab80 2023-12-25T21:06:33.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.845+0000 7f7e2f7fe700 1 --2- 172.21.15.61:0/3653678999 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f7e1c02fd50 0x7f7e1c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:33.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.845+0000 7f7e36948700 1 -- 172.21.15.61:0/3653678999 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e30059100 con 0x7f7e300fab80 2023-12-25T21:06:33.848 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.845+0000 7f7e2f7fe700 1 --2- 172.21.15.61:0/3653678999 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f7e1c02fd50 0x7f7e1c032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f7e240096a0 tx=0x7f7e24006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:33.850 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:33.849+0000 7f7e2cff9700 1 -- 172.21.15.61:0/3653678999 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e20016070 con 0x7f7e300fab80 2023-12-25T21:06:34.034 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.033+0000 7f7e36948700 1 -- 172.21.15.61:0/3653678999 --> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e300fc900 con 0x7f7e1c02fd50 2023-12-25T21:06:34.038 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.037+0000 7f7e2cff9700 1 -- 172.21.15.61:0/3653678999 <== mgr.14116 v2:172.21.15.61:6800/19001851 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f7e300fc900 con 0x7f7e1c02fd50 2023-12-25T21:06:34.038 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: Scheduled crash update... 2023-12-25T21:06:34.040 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.039+0000 7f7e36948700 1 -- 172.21.15.61:0/3653678999 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f7e1c02fd50 msgr2=0x7f7e1c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:34.040 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.039+0000 7f7e36948700 1 --2- 172.21.15.61:0/3653678999 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f7e1c02fd50 0x7f7e1c032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f7e240096a0 tx=0x7f7e24006b40).stop 2023-12-25T21:06:34.040 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.039+0000 7f7e36948700 1 -- 172.21.15.61:0/3653678999 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e300fab80 msgr2=0x7f7e3010d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:34.041 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.039+0000 7f7e36948700 1 --2- 172.21.15.61:0/3653678999 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e300fab80 0x7f7e3010d740 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f7e2000e040 tx=0x7f7e20006ee0).stop 2023-12-25T21:06:34.041 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.039+0000 7f7e36948700 1 -- 172.21.15.61:0/3653678999 shutdown_connections 2023-12-25T21:06:34.041 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.039+0000 7f7e36948700 1 --2- 172.21.15.61:0/3653678999 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f7e1c02fd50 0x7f7e1c032210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:34.041 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.039+0000 7f7e36948700 1 --2- 172.21.15.61:0/3653678999 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e300fab80 0x7f7e3010d740 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:34.041 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.039+0000 7f7e36948700 1 -- 172.21.15.61:0/3653678999 >> 172.21.15.61:0/3653678999 conn(0x7f7e300f63c0 msgr2=0x7f7e300f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:34.041 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.040+0000 7f7e36948700 1 -- 172.21.15.61:0/3653678999 shutdown_connections 2023-12-25T21:06:34.041 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.040+0000 7f7e36948700 1 -- 172.21.15.61:0/3653678999 wait complete. 2023-12-25T21:06:34.455 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:34 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:33.115938+0000 mgr.smithi061.jxxeke (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T21:06:34.455 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:34 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cephadm 2023-12-25T21:06:33.116479+0000 mgr.smithi061.jxxeke (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-12-25T21:06:34.456 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:34 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:34.038257+0000 mon.smithi061 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.61:0/2770331949' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:34.462 INFO:teuthology.orchestra.run.smithi061.stderr:Enabling mgr prometheus module... 2023-12-25T21:06:34.462 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2023-12-25T21:06:34.839 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.836+0000 7f8d5c8c2700 1 Processor -- start 2023-12-25T21:06:34.839 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.836+0000 7f8d5c8c2700 1 -- start start 2023-12-25T21:06:34.839 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.837+0000 7f8d5c8c2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d580703d0 0x7f8d580fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:34.840 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.837+0000 7f8d5c8c2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d580fee30 con 0x7f8d580703d0 2023-12-25T21:06:34.840 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.837+0000 7f8d5659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d580703d0 0x7f8d580fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:34.840 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.837+0000 7f8d5659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d580703d0 0x7f8d580fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59266/0 (socket says 172.21.15.61:59266) 2023-12-25T21:06:34.840 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.837+0000 7f8d5659c700 1 -- 172.21.15.61:0/497332158 learned_addr learned my addr 172.21.15.61:0/497332158 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:34.840 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.837+0000 7f8d5659c700 1 -- 172.21.15.61:0/497332158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d580fef70 con 0x7f8d580703d0 2023-12-25T21:06:34.840 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.837+0000 7f8d5659c700 1 --2- 172.21.15.61:0/497332158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d580703d0 0x7f8d580fe940 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f8d48009960 tx=0x7f8d480092d0).ready entity=mon.0 client_cookie=70f392b0ca7ab645 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:34.840 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.837+0000 7f8d5559a700 1 -- 172.21.15.61:0/497332158 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d4800b9c0 con 0x7f8d580703d0 2023-12-25T21:06:34.840 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.837+0000 7f8d5559a700 1 -- 172.21.15.61:0/497332158 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8d48011400 con 0x7f8d580703d0 2023-12-25T21:06:34.840 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.837+0000 7f8d5559a700 1 -- 172.21.15.61:0/497332158 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d480103a0 con 0x7f8d580703d0 2023-12-25T21:06:34.840 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.838+0000 7f8d5c8c2700 1 -- 172.21.15.61:0/497332158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d580703d0 msgr2=0x7f8d580fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:34.840 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.838+0000 7f8d5c8c2700 1 --2- 172.21.15.61:0/497332158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d580703d0 0x7f8d580fe940 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f8d48009960 tx=0x7f8d480092d0).stop 2023-12-25T21:06:34.840 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.838+0000 7f8d5c8c2700 1 -- 172.21.15.61:0/497332158 shutdown_connections 2023-12-25T21:06:34.840 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.838+0000 7f8d5c8c2700 1 --2- 172.21.15.61:0/497332158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d580703d0 0x7f8d580fe940 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:34.841 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.838+0000 7f8d5c8c2700 1 -- 172.21.15.61:0/497332158 >> 172.21.15.61:0/497332158 conn(0x7f8d580f63c0 msgr2=0x7f8d580f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:34.841 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.838+0000 7f8d5c8c2700 1 -- 172.21.15.61:0/497332158 shutdown_connections 2023-12-25T21:06:34.841 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.838+0000 7f8d5c8c2700 1 -- 172.21.15.61:0/497332158 wait complete. 2023-12-25T21:06:34.841 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.839+0000 7f8d5c8c2700 1 Processor -- start 2023-12-25T21:06:34.841 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.839+0000 7f8d5c8c2700 1 -- start start 2023-12-25T21:06:34.841 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.839+0000 7f8d5c8c2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d580703d0 0x7f8d5810a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:34.841 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.839+0000 7f8d5c8c2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5810aee0 con 0x7f8d580703d0 2023-12-25T21:06:34.843 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.839+0000 7f8d5659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d580703d0 0x7f8d5810a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:34.843 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.839+0000 7f8d5659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d580703d0 0x7f8d5810a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59282/0 (socket says 172.21.15.61:59282) 2023-12-25T21:06:34.843 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.839+0000 7f8d5659c700 1 -- 172.21.15.61:0/2360057295 learned_addr learned my addr 172.21.15.61:0/2360057295 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:34.843 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.839+0000 7f8d5659c700 1 -- 172.21.15.61:0/2360057295 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d48009570 con 0x7f8d580703d0 2023-12-25T21:06:34.843 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.839+0000 7f8d5659c700 1 --2- 172.21.15.61:0/2360057295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d580703d0 0x7f8d5810a9f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f8d48000f80 tx=0x7f8d480092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:34.844 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.840+0000 7f8d4effd700 1 -- 172.21.15.61:0/2360057295 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d48010820 con 0x7f8d580703d0 2023-12-25T21:06:34.844 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.840+0000 7f8d4effd700 1 -- 172.21.15.61:0/2360057295 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8d48010df0 con 0x7f8d580703d0 2023-12-25T21:06:34.844 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.840+0000 7f8d5c8c2700 1 -- 172.21.15.61:0/2360057295 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5810b0e0 con 0x7f8d580703d0 2023-12-25T21:06:34.844 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.840+0000 7f8d4effd700 1 -- 172.21.15.61:0/2360057295 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d48023410 con 0x7f8d580703d0 2023-12-25T21:06:34.844 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.840+0000 7f8d5c8c2700 1 -- 172.21.15.61:0/2360057295 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5810b4f0 con 0x7f8d580703d0 2023-12-25T21:06:34.844 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.841+0000 7f8d4effd700 1 -- 172.21.15.61:0/2360057295 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f8d4800f070 con 0x7f8d580703d0 2023-12-25T21:06:34.844 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.841+0000 7f8d4effd700 1 --2- 172.21.15.61:0/2360057295 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f8d3802fdb0 0x7f8d38032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:34.844 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.841+0000 7f8d5c8c2700 1 -- 172.21.15.61:0/2360057295 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d58045ad0 con 0x7f8d580703d0 2023-12-25T21:06:34.844 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.841+0000 7f8d4effd700 1 -- 172.21.15.61:0/2360057295 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f8d48041e60 con 0x7f8d580703d0 2023-12-25T21:06:34.844 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.841+0000 7f8d55d9b700 1 --2- 172.21.15.61:0/2360057295 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f8d3802fdb0 0x7f8d38032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:34.844 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.841+0000 7f8d55d9b700 1 --2- 172.21.15.61:0/2360057295 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f8d3802fdb0 0x7f8d38032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f8d400096a0 tx=0x7f8d40006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:34.846 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:34.845+0000 7f8d4effd700 1 -- 172.21.15.61:0/2360057295 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d4801e020 con 0x7f8d580703d0 2023-12-25T21:06:35.044 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:35.044+0000 7f8d5c8c2700 1 -- 172.21.15.61:0/2360057295 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f8d580ffb90 con 0x7f8d580703d0 2023-12-25T21:06:35.275 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:35.275+0000 7f8d4effd700 1 -- 172.21.15.61:0/2360057295 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f8d4802c3f0 con 0x7f8d580703d0 2023-12-25T21:06:35.277 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:35.276+0000 7f8d4effd700 1 -- 172.21.15.61:0/2360057295 <== mon.0 v2:172.21.15.61:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f8d48022760 con 0x7f8d580703d0 2023-12-25T21:06:35.280 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:35.278+0000 7f8d5c8c2700 1 -- 172.21.15.61:0/2360057295 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f8d3802fdb0 msgr2=0x7f8d38032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:35.280 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:35.278+0000 7f8d5c8c2700 1 --2- 172.21.15.61:0/2360057295 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f8d3802fdb0 0x7f8d38032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f8d400096a0 tx=0x7f8d40006b40).stop 2023-12-25T21:06:35.280 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:35.278+0000 7f8d5c8c2700 1 -- 172.21.15.61:0/2360057295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d580703d0 msgr2=0x7f8d5810a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:35.280 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:35.278+0000 7f8d5c8c2700 1 --2- 172.21.15.61:0/2360057295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d580703d0 0x7f8d5810a9f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f8d48000f80 tx=0x7f8d480092d0).stop 2023-12-25T21:06:35.280 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:35.278+0000 7f8d5c8c2700 1 -- 172.21.15.61:0/2360057295 shutdown_connections 2023-12-25T21:06:35.280 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:35.278+0000 7f8d5c8c2700 1 --2- 172.21.15.61:0/2360057295 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7f8d3802fdb0 0x7f8d38032270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:35.280 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:35.278+0000 7f8d5c8c2700 1 --2- 172.21.15.61:0/2360057295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d580703d0 0x7f8d5810a9f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:35.280 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:35.278+0000 7f8d5c8c2700 1 -- 172.21.15.61:0/2360057295 >> 172.21.15.61:0/2360057295 conn(0x7f8d580f63c0 msgr2=0x7f8d580f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:35.281 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:35.279+0000 7f8d5c8c2700 1 -- 172.21.15.61:0/2360057295 shutdown_connections 2023-12-25T21:06:35.281 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:35.279+0000 7f8d5c8c2700 1 -- 172.21.15.61:0/2360057295 wait complete. 2023-12-25T21:06:35.606 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:35 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:34.035508+0000 mgr.smithi061.jxxeke (mgr.14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T21:06:35.606 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:35 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cephadm 2023-12-25T21:06:34.036152+0000 mgr.smithi061.jxxeke (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2023-12-25T21:06:35.606 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:35 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:35.045452+0000 mon.smithi061 (mon.0) 60 : audit [INF] from='client.? 172.21.15.61:0/2360057295' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-12-25T21:06:35.646 INFO:teuthology.orchestra.run.smithi061.stderr:Deploying prometheus service with default placement... 2023-12-25T21:06:35.646 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2023-12-25T21:06:36.022 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.020+0000 7feed9b60700 1 Processor -- start 2023-12-25T21:06:36.022 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.020+0000 7feed9b60700 1 -- start start 2023-12-25T21:06:36.022 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.020+0000 7feed9b60700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feed4070430 0x7feed40fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:36.022 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.020+0000 7feed9b60700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed40fec20 con 0x7feed4070430 2023-12-25T21:06:36.022 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.020+0000 7feed8b5e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feed4070430 0x7feed40fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:36.022 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.020+0000 7feed8b5e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feed4070430 0x7feed40fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59310/0 (socket says 172.21.15.61:59310) 2023-12-25T21:06:36.022 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.020+0000 7feed8b5e700 1 -- 172.21.15.61:0/4183550165 learned_addr learned my addr 172.21.15.61:0/4183550165 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:36.023 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.021+0000 7feed8b5e700 1 -- 172.21.15.61:0/4183550165 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed40fed60 con 0x7feed4070430 2023-12-25T21:06:36.023 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.021+0000 7feed8b5e700 1 --2- 172.21.15.61:0/4183550165 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feed4070430 0x7feed40fe730 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7feec800b5f0 tx=0x7feec8006ee0).ready entity=mon.0 client_cookie=5c4b6121c07fdf19 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:36.023 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.021+0000 7feed37fe700 1 -- 172.21.15.61:0/4183550165 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feec800d600 con 0x7feed4070430 2023-12-25T21:06:36.023 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.021+0000 7feed37fe700 1 -- 172.21.15.61:0/4183550165 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7feec800dbd0 con 0x7feed4070430 2023-12-25T21:06:36.023 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.021+0000 7feed37fe700 1 -- 172.21.15.61:0/4183550165 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feec8011c00 con 0x7feed4070430 2023-12-25T21:06:36.024 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.022+0000 7feed9b60700 1 -- 172.21.15.61:0/4183550165 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feed4070430 msgr2=0x7feed40fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:36.024 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.022+0000 7feed9b60700 1 --2- 172.21.15.61:0/4183550165 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feed4070430 0x7feed40fe730 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7feec800b5f0 tx=0x7feec8006ee0).stop 2023-12-25T21:06:36.024 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.022+0000 7feed9b60700 1 -- 172.21.15.61:0/4183550165 shutdown_connections 2023-12-25T21:06:36.024 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.022+0000 7feed9b60700 1 --2- 172.21.15.61:0/4183550165 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feed4070430 0x7feed40fe730 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:36.024 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.022+0000 7feed9b60700 1 -- 172.21.15.61:0/4183550165 >> 172.21.15.61:0/4183550165 conn(0x7feed40f6210 msgr2=0x7feed40f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:36.024 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.022+0000 7feed9b60700 1 -- 172.21.15.61:0/4183550165 shutdown_connections 2023-12-25T21:06:36.024 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.023+0000 7feed9b60700 1 -- 172.21.15.61:0/4183550165 wait complete. 2023-12-25T21:06:36.025 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.024+0000 7feed9b60700 1 Processor -- start 2023-12-25T21:06:36.025 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.024+0000 7feed9b60700 1 -- start start 2023-12-25T21:06:36.026 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.024+0000 7feed9b60700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feed4070430 0x7feed40fc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:36.026 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.024+0000 7feed9b60700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed40fa630 con 0x7feed4070430 2023-12-25T21:06:36.026 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.024+0000 7feed8b5e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feed4070430 0x7feed40fc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:36.026 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.024+0000 7feed8b5e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feed4070430 0x7feed40fc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59314/0 (socket says 172.21.15.61:59314) 2023-12-25T21:06:36.026 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.024+0000 7feed8b5e700 1 -- 172.21.15.61:0/545942453 learned_addr learned my addr 172.21.15.61:0/545942453 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:36.026 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.024+0000 7feed8b5e700 1 -- 172.21.15.61:0/545942453 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec800b200 con 0x7feed4070430 2023-12-25T21:06:36.026 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.025+0000 7feed8b5e700 1 --2- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feed4070430 0x7feed40fc1b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7feec800b190 tx=0x7feec8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:36.026 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.025+0000 7feed17fa700 1 -- 172.21.15.61:0/545942453 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feec801b410 con 0x7feed4070430 2023-12-25T21:06:36.026 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.025+0000 7feed9b60700 1 -- 172.21.15.61:0/545942453 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed40fa7d0 con 0x7feed4070430 2023-12-25T21:06:36.026 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.025+0000 7feed17fa700 1 -- 172.21.15.61:0/545942453 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7feec801b9e0 con 0x7feed4070430 2023-12-25T21:06:36.027 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.025+0000 7feed17fa700 1 -- 172.21.15.61:0/545942453 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feec801a880 con 0x7feed4070430 2023-12-25T21:06:36.027 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.025+0000 7feed9b60700 1 -- 172.21.15.61:0/545942453 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed40fabe0 con 0x7feed4070430 2023-12-25T21:06:36.027 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.026+0000 7feed17fa700 1 -- 172.21.15.61:0/545942453 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7feec800f070 con 0x7feed4070430 2023-12-25T21:06:36.028 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.026+0000 7feed17fa700 1 --2- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7feeb802fdb0 0x7feeb8032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:36.028 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.026+0000 7feed9b60700 1 -- 172.21.15.61:0/545942453 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed4045ad0 con 0x7feed4070430 2023-12-25T21:06:36.028 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.026+0000 7feed3fff700 1 -- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7feeb802fdb0 msgr2=0x7feeb8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/19001851 2023-12-25T21:06:36.028 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.026+0000 7feed17fa700 1 -- 172.21.15.61:0/545942453 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7feec8041340 con 0x7feed4070430 2023-12-25T21:06:36.028 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.026+0000 7feed3fff700 1 --2- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7feeb802fdb0 0x7feeb8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-25T21:06:36.034 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.033+0000 7feed17fa700 1 -- 172.21.15.61:0/545942453 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feec8010360 con 0x7feed4070430 2023-12-25T21:06:36.220 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.219+0000 7feed9b60700 1 -- 172.21.15.61:0/545942453 --> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7feed40fae20 con 0x7feeb802fdb0 2023-12-25T21:06:36.228 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.227+0000 7feed3fff700 1 -- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7feeb802fdb0 msgr2=0x7feeb8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/19001851 2023-12-25T21:06:36.228 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.227+0000 7feed3fff700 1 --2- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7feeb802fdb0 0x7feeb8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-25T21:06:36.628 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.628+0000 7feed3fff700 1 -- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7feeb802fdb0 msgr2=0x7feeb8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/19001851 2023-12-25T21:06:36.629 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:36.628+0000 7feed3fff700 1 --2- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7feeb802fdb0 0x7feeb8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-25T21:06:36.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:36 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:35.277277+0000 mon.smithi061 (mon.0) 61 : audit [INF] from='client.? 172.21.15.61:0/2360057295' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-12-25T21:06:36.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:36 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:35.277329+0000 mon.smithi061 (mon.0) 62 : cluster [DBG] mgrmap e9: smithi061.jxxeke(active, since 16s) 2023-12-25T21:06:37.429 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:37.429+0000 7feed3fff700 1 -- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7feeb802fdb0 msgr2=0x7feeb8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/19001851 2023-12-25T21:06:37.430 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:37.429+0000 7feed3fff700 1 --2- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7feeb802fdb0 0x7feeb8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-12-25T21:06:39.031 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:39.030+0000 7feed3fff700 1 -- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7feeb802fdb0 msgr2=0x7feeb8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/19001851 2023-12-25T21:06:39.031 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:39.030+0000 7feed3fff700 1 --2- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7feeb802fdb0 0x7feeb8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-12-25T21:06:40.499 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:40.498+0000 7feed17fa700 1 -- 172.21.15.61:0/545942453 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7feec8024ca0 con 0x7feed4070430 2023-12-25T21:06:40.499 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:40.498+0000 7feed17fa700 1 -- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7feeb802fdb0 msgr2=0x7feeb8032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-25T21:06:40.499 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:40.498+0000 7feed17fa700 1 --2- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7feeb802fdb0 0x7feeb8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T21:06:40.754 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:40 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:40.443862+0000 mon.smithi061 (mon.0) 63 : cluster [INF] Active manager daemon smithi061.jxxeke restarted 2023-12-25T21:06:40.754 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:40 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:40.444243+0000 mon.smithi061 (mon.0) 64 : cluster [INF] Activating manager daemon smithi061.jxxeke 2023-12-25T21:06:40.754 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:40 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:40.448411+0000 mon.smithi061 (mon.0) 65 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-12-25T21:06:41.507 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:41.505+0000 7feed17fa700 1 -- 172.21.15.61:0/545942453 <== mon.0 v2:172.21.15.61:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7feec803faf0 con 0x7feed4070430 2023-12-25T21:06:41.508 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:41.505+0000 7feed17fa700 1 --2- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7feeb8033c20 0x7feeb8036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:41.508 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:41.505+0000 7feed17fa700 1 -- 172.21.15.61:0/545942453 --> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7feed40fae20 con 0x7feeb8033c20 2023-12-25T21:06:41.508 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:41.506+0000 7feed3fff700 1 --2- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7feeb8033c20 0x7feeb8036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:41.508 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:41.506+0000 7feed3fff700 1 --2- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7feeb8033c20 0x7feeb8036010 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7feec00036d0 tx=0x7feec0006cb0).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:41.515 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:41.514+0000 7feed17fa700 1 -- 172.21.15.61:0/545942453 <== mgr.14144 v2:172.21.15.61:6800/2156288800 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7feed40fae20 con 0x7feeb8033c20 2023-12-25T21:06:41.515 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-12-25T21:06:41.521 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:41.516+0000 7feed9b60700 1 -- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7feeb8033c20 msgr2=0x7feeb8036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:41.521 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:41.516+0000 7feed9b60700 1 --2- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7feeb8033c20 0x7feeb8036010 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7feec00036d0 tx=0x7feec0006cb0).stop 2023-12-25T21:06:41.522 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:41.516+0000 7feed9b60700 1 -- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feed4070430 msgr2=0x7feed40fc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:41.522 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:41.516+0000 7feed9b60700 1 --2- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feed4070430 0x7feed40fc1b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7feec800b190 tx=0x7feec8006ee0).stop 2023-12-25T21:06:41.522 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:41.516+0000 7feed9b60700 1 -- 172.21.15.61:0/545942453 shutdown_connections 2023-12-25T21:06:41.522 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:41.516+0000 7feed9b60700 1 --2- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7feeb8033c20 0x7feeb8036010 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:41.522 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:41.516+0000 7feed9b60700 1 --2- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feed4070430 0x7feed40fc1b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:41.522 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:41.516+0000 7feed9b60700 1 --2- 172.21.15.61:0/545942453 >> [v2:172.21.15.61:6800/19001851,v1:172.21.15.61:6801/19001851] conn(0x7feeb802fdb0 0x7feeb8032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T21:06:41.522 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:41.516+0000 7feed9b60700 1 -- 172.21.15.61:0/545942453 >> 172.21.15.61:0/545942453 conn(0x7feed40f6210 msgr2=0x7feed40f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:41.522 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:41.516+0000 7feed9b60700 1 -- 172.21.15.61:0/545942453 shutdown_connections 2023-12-25T21:06:41.522 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:41.516+0000 7feed9b60700 1 -- 172.21.15.61:0/545942453 wait complete. 2023-12-25T21:06:41.862 INFO:teuthology.orchestra.run.smithi061.stderr:Deploying grafana service with default placement... 2023-12-25T21:06:41.862 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2023-12-25T21:06:41.863 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:41 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:40.500180+0000 mon.smithi061 (mon.0) 66 : cluster [DBG] mgrmap e10: smithi061.jxxeke(active, starting, since 0.0560187s) 2023-12-25T21:06:41.864 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:41 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:40.501785+0000 mon.smithi061 (mon.0) 67 : audit [DBG] from='mgr.14144 172.21.15.61:0/408974420' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "mon metadata", "id": "smithi061"}]: dispatch 2023-12-25T21:06:41.864 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:41 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:40.502391+0000 mon.smithi061 (mon.0) 68 : audit [DBG] from='mgr.14144 172.21.15.61:0/408974420' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "mgr metadata", "who": "smithi061.jxxeke", "id": "smithi061.jxxeke"}]: dispatch 2023-12-25T21:06:41.864 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:41 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:40.503238+0000 mon.smithi061 (mon.0) 69 : audit [DBG] from='mgr.14144 172.21.15.61:0/408974420' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-25T21:06:41.864 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:41 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:40.503380+0000 mon.smithi061 (mon.0) 70 : audit [DBG] from='mgr.14144 172.21.15.61:0/408974420' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-25T21:06:41.864 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:41 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:40.503491+0000 mon.smithi061 (mon.0) 71 : audit [DBG] from='mgr.14144 172.21.15.61:0/408974420' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-25T21:06:41.864 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:41 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:40.518612+0000 mon.smithi061 (mon.0) 72 : cluster [INF] Manager daemon smithi061.jxxeke is now available 2023-12-25T21:06:41.864 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:41 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:40.621338+0000 mon.smithi061 (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.61:0/408974420' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-25T21:06:41.864 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:41 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:40.622692+0000 mon.smithi061 (mon.0) 74 : audit [DBG] from='mgr.14144 172.21.15.61:0/408974420' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-25T21:06:41.864 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:41 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:40.639362+0000 mon.smithi061 (mon.0) 75 : audit [INF] from='mgr.14144 172.21.15.61:0/408974420' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/mirror_snapshot_schedule"}]: dispatch 2023-12-25T21:06:41.864 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:41 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:40.647176+0000 mon.smithi061 (mon.0) 76 : audit [INF] from='mgr.14144 172.21.15.61:0/408974420' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/trash_purge_schedule"}]: dispatch 2023-12-25T21:06:42.237 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.236+0000 7f8a62329700 1 Processor -- start 2023-12-25T21:06:42.238 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.237+0000 7f8a62329700 1 -- start start 2023-12-25T21:06:42.238 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.237+0000 7f8a62329700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a5c0703d0 0x7f8a5c0fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:42.238 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.237+0000 7f8a62329700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c0fee30 con 0x7f8a5c0703d0 2023-12-25T21:06:42.238 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.237+0000 7f8a5bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a5c0703d0 0x7f8a5c0fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:42.241 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.237+0000 7f8a5bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a5c0703d0 0x7f8a5c0fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:36350/0 (socket says 172.21.15.61:36350) 2023-12-25T21:06:42.241 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.237+0000 7f8a5bfff700 1 -- 172.21.15.61:0/948603938 learned_addr learned my addr 172.21.15.61:0/948603938 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:42.241 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.238+0000 7f8a5bfff700 1 -- 172.21.15.61:0/948603938 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c0fef70 con 0x7f8a5c0703d0 2023-12-25T21:06:42.241 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.238+0000 7f8a5bfff700 1 --2- 172.21.15.61:0/948603938 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a5c0703d0 0x7f8a5c0fe940 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f8a4c00b5f0 tx=0x7f8a4c006ee0).ready entity=mon.0 client_cookie=71a341176384a7f server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:42.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.238+0000 7f8a5affd700 1 -- 172.21.15.61:0/948603938 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a4c00d600 con 0x7f8a5c0703d0 2023-12-25T21:06:42.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.238+0000 7f8a5affd700 1 -- 172.21.15.61:0/948603938 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8a4c00dbd0 con 0x7f8a5c0703d0 2023-12-25T21:06:42.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.238+0000 7f8a5affd700 1 -- 172.21.15.61:0/948603938 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a4c011c00 con 0x7f8a5c0703d0 2023-12-25T21:06:42.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.239+0000 7f8a62329700 1 -- 172.21.15.61:0/948603938 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a5c0703d0 msgr2=0x7f8a5c0fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:42.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.239+0000 7f8a62329700 1 --2- 172.21.15.61:0/948603938 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a5c0703d0 0x7f8a5c0fe940 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f8a4c00b5f0 tx=0x7f8a4c006ee0).stop 2023-12-25T21:06:42.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.239+0000 7f8a62329700 1 -- 172.21.15.61:0/948603938 shutdown_connections 2023-12-25T21:06:42.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.239+0000 7f8a62329700 1 --2- 172.21.15.61:0/948603938 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a5c0703d0 0x7f8a5c0fe940 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:42.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.239+0000 7f8a62329700 1 -- 172.21.15.61:0/948603938 >> 172.21.15.61:0/948603938 conn(0x7f8a5c0f63c0 msgr2=0x7f8a5c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:42.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.239+0000 7f8a62329700 1 -- 172.21.15.61:0/948603938 shutdown_connections 2023-12-25T21:06:42.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.239+0000 7f8a62329700 1 -- 172.21.15.61:0/948603938 wait complete. 2023-12-25T21:06:42.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.239+0000 7f8a62329700 1 Processor -- start 2023-12-25T21:06:42.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.240+0000 7f8a62329700 1 -- start start 2023-12-25T21:06:42.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.240+0000 7f8a62329700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a5c0703d0 0x7f8a5c0fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:42.242 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.240+0000 7f8a62329700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c0fa840 con 0x7f8a5c0703d0 2023-12-25T21:06:42.243 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.240+0000 7f8a5bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a5c0703d0 0x7f8a5c0fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:42.243 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.240+0000 7f8a5bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a5c0703d0 0x7f8a5c0fc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:36358/0 (socket says 172.21.15.61:36358) 2023-12-25T21:06:42.243 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.240+0000 7f8a5bfff700 1 -- 172.21.15.61:0/1876659109 learned_addr learned my addr 172.21.15.61:0/1876659109 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:42.243 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.240+0000 7f8a5bfff700 1 -- 172.21.15.61:0/1876659109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4c00b200 con 0x7f8a5c0703d0 2023-12-25T21:06:42.243 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.240+0000 7f8a5bfff700 1 --2- 172.21.15.61:0/1876659109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a5c0703d0 0x7f8a5c0fc3c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f8a4c00b190 tx=0x7f8a4c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:42.243 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.240+0000 7f8a58ff9700 1 -- 172.21.15.61:0/1876659109 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a4c01b410 con 0x7f8a5c0703d0 2023-12-25T21:06:42.243 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.240+0000 7f8a58ff9700 1 -- 172.21.15.61:0/1876659109 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8a4c01b9e0 con 0x7f8a5c0703d0 2023-12-25T21:06:42.243 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.240+0000 7f8a62329700 1 -- 172.21.15.61:0/1876659109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a5c0fa9e0 con 0x7f8a5c0703d0 2023-12-25T21:06:42.243 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.240+0000 7f8a58ff9700 1 -- 172.21.15.61:0/1876659109 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a4c01a880 con 0x7f8a5c0703d0 2023-12-25T21:06:42.243 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.240+0000 7f8a62329700 1 -- 172.21.15.61:0/1876659109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a5c0fadf0 con 0x7f8a5c0703d0 2023-12-25T21:06:42.244 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.241+0000 7f8a58ff9700 1 -- 172.21.15.61:0/1876659109 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f8a4c00f070 con 0x7f8a5c0703d0 2023-12-25T21:06:42.244 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.241+0000 7f8a58ff9700 1 --2- 172.21.15.61:0/1876659109 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f8a4802fcf0 0x7f8a480321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:42.244 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.241+0000 7f8a58ff9700 1 -- 172.21.15.61:0/1876659109 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f8a4c0412b0 con 0x7f8a5c0703d0 2023-12-25T21:06:42.244 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.241+0000 7f8a62329700 1 -- 172.21.15.61:0/1876659109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a5c045ad0 con 0x7f8a5c0703d0 2023-12-25T21:06:42.244 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.242+0000 7f8a5b7fe700 1 --2- 172.21.15.61:0/1876659109 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f8a4802fcf0 0x7f8a480321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:42.244 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.242+0000 7f8a5b7fe700 1 --2- 172.21.15.61:0/1876659109 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f8a4802fcf0 0x7f8a480321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8a500096a0 tx=0x7f8a50006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:42.247 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.246+0000 7f8a58ff9700 1 -- 172.21.15.61:0/1876659109 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a4c010360 con 0x7f8a5c0703d0 2023-12-25T21:06:42.435 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.434+0000 7f8a62329700 1 -- 172.21.15.61:0/1876659109 --> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a5c104890 con 0x7f8a4802fcf0 2023-12-25T21:06:42.438 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.438+0000 7f8a58ff9700 1 -- 172.21.15.61:0/1876659109 <== mgr.14144 v2:172.21.15.61:6800/2156288800 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f8a5c104890 con 0x7f8a4802fcf0 2023-12-25T21:06:42.438 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-12-25T21:06:42.441 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.440+0000 7f8a62329700 1 -- 172.21.15.61:0/1876659109 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f8a4802fcf0 msgr2=0x7f8a480321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:42.441 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.440+0000 7f8a62329700 1 --2- 172.21.15.61:0/1876659109 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f8a4802fcf0 0x7f8a480321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8a500096a0 tx=0x7f8a50006b40).stop 2023-12-25T21:06:42.441 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.440+0000 7f8a62329700 1 -- 172.21.15.61:0/1876659109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a5c0703d0 msgr2=0x7f8a5c0fc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:42.442 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.440+0000 7f8a62329700 1 --2- 172.21.15.61:0/1876659109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a5c0703d0 0x7f8a5c0fc3c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f8a4c00b190 tx=0x7f8a4c006ee0).stop 2023-12-25T21:06:42.442 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.440+0000 7f8a62329700 1 -- 172.21.15.61:0/1876659109 shutdown_connections 2023-12-25T21:06:42.442 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.440+0000 7f8a62329700 1 --2- 172.21.15.61:0/1876659109 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f8a4802fcf0 0x7f8a480321b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:42.442 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.440+0000 7f8a62329700 1 --2- 172.21.15.61:0/1876659109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a5c0703d0 0x7f8a5c0fc3c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:42.442 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.440+0000 7f8a62329700 1 -- 172.21.15.61:0/1876659109 >> 172.21.15.61:0/1876659109 conn(0x7f8a5c0f63c0 msgr2=0x7f8a5c0f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:42.442 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.440+0000 7f8a62329700 1 -- 172.21.15.61:0/1876659109 shutdown_connections 2023-12-25T21:06:42.442 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:42.440+0000 7f8a62329700 1 -- 172.21.15.61:0/1876659109 wait complete. 2023-12-25T21:06:42.730 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:42 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:41.508026+0000 mon.smithi061 (mon.0) 77 : cluster [DBG] mgrmap e11: smithi061.jxxeke(active, since 1.06387s) 2023-12-25T21:06:42.730 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:42 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cephadm 2023-12-25T21:06:41.509459+0000 mgr.smithi061.jxxeke (mgr.14144) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-12-25T21:06:42.730 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:42 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:41.515143+0000 mon.smithi061 (mon.0) 78 : audit [INF] from='mgr.14144 172.21.15.61:0/408974420' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:42.730 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:42 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:42.438724+0000 mon.smithi061 (mon.0) 79 : audit [INF] from='mgr.14144 172.21.15.61:0/408974420' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:42.786 INFO:teuthology.orchestra.run.smithi061.stderr:Deploying node-exporter service with default placement... 2023-12-25T21:06:42.786 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2023-12-25T21:06:43.272 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.269+0000 7f6284e18700 1 Processor -- start 2023-12-25T21:06:43.273 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.269+0000 7f6284e18700 1 -- start start 2023-12-25T21:06:43.273 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.269+0000 7f6284e18700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62800703c0 0x7f62800fe9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:43.273 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.269+0000 7f6284e18700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62800707e0 con 0x7f62800703c0 2023-12-25T21:06:43.273 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.270+0000 7f627e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62800703c0 0x7f62800fe9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:43.273 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.270+0000 7f627e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62800703c0 0x7f62800fe9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:36372/0 (socket says 172.21.15.61:36372) 2023-12-25T21:06:43.273 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.270+0000 7f627e59c700 1 -- 172.21.15.61:0/558881220 learned_addr learned my addr 172.21.15.61:0/558881220 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:43.273 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.270+0000 7f627e59c700 1 -- 172.21.15.61:0/558881220 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62800feec0 con 0x7f62800703c0 2023-12-25T21:06:43.273 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.270+0000 7f627e59c700 1 --2- 172.21.15.61:0/558881220 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62800703c0 0x7f62800fe9d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f627400b5f0 tx=0x7f6274006ee0).ready entity=mon.0 client_cookie=56cdd152699d18fc server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:43.273 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.270+0000 7f627d59a700 1 -- 172.21.15.61:0/558881220 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f627400d600 con 0x7f62800703c0 2023-12-25T21:06:43.273 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.270+0000 7f627d59a700 1 -- 172.21.15.61:0/558881220 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f627400dbd0 con 0x7f62800703c0 2023-12-25T21:06:43.273 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.270+0000 7f627d59a700 1 -- 172.21.15.61:0/558881220 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6274011c00 con 0x7f62800703c0 2023-12-25T21:06:43.274 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.271+0000 7f6284e18700 1 -- 172.21.15.61:0/558881220 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62800703c0 msgr2=0x7f62800fe9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:43.274 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.271+0000 7f6284e18700 1 --2- 172.21.15.61:0/558881220 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62800703c0 0x7f62800fe9d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f627400b5f0 tx=0x7f6274006ee0).stop 2023-12-25T21:06:43.274 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.271+0000 7f6284e18700 1 -- 172.21.15.61:0/558881220 shutdown_connections 2023-12-25T21:06:43.274 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.271+0000 7f6284e18700 1 --2- 172.21.15.61:0/558881220 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62800703c0 0x7f62800fe9d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:43.274 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.271+0000 7f6284e18700 1 -- 172.21.15.61:0/558881220 >> 172.21.15.61:0/558881220 conn(0x7f62800f63b0 msgr2=0x7f62800f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:43.274 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.271+0000 7f6284e18700 1 -- 172.21.15.61:0/558881220 shutdown_connections 2023-12-25T21:06:43.274 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.271+0000 7f6284e18700 1 -- 172.21.15.61:0/558881220 wait complete. 2023-12-25T21:06:43.274 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.272+0000 7f6284e18700 1 Processor -- start 2023-12-25T21:06:43.274 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.272+0000 7f6284e18700 1 -- start start 2023-12-25T21:06:43.277 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.272+0000 7f6284e18700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62800703c0 0x7f628010a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:43.277 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.272+0000 7f6284e18700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628010aec0 con 0x7f62800703c0 2023-12-25T21:06:43.277 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.272+0000 7f627e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62800703c0 0x7f628010a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:43.277 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.272+0000 7f627e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62800703c0 0x7f628010a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:36384/0 (socket says 172.21.15.61:36384) 2023-12-25T21:06:43.277 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.272+0000 7f627e59c700 1 -- 172.21.15.61:0/3759814433 learned_addr learned my addr 172.21.15.61:0/3759814433 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:43.278 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.273+0000 7f627e59c700 1 -- 172.21.15.61:0/3759814433 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627400b200 con 0x7f62800703c0 2023-12-25T21:06:43.278 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.273+0000 7f627e59c700 1 --2- 172.21.15.61:0/3759814433 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62800703c0 0x7f628010a9d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f627400e040 tx=0x7f6274006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:43.278 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.273+0000 7f626effd700 1 -- 172.21.15.61:0/3759814433 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f627400d9f0 con 0x7f62800703c0 2023-12-25T21:06:43.278 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.273+0000 7f626effd700 1 -- 172.21.15.61:0/3759814433 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f627401a9f0 con 0x7f62800703c0 2023-12-25T21:06:43.278 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.273+0000 7f6284e18700 1 -- 172.21.15.61:0/3759814433 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628010b0c0 con 0x7f62800703c0 2023-12-25T21:06:43.278 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.273+0000 7f626effd700 1 -- 172.21.15.61:0/3759814433 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f627401b890 con 0x7f62800703c0 2023-12-25T21:06:43.278 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.273+0000 7f6284e18700 1 -- 172.21.15.61:0/3759814433 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628010b4d0 con 0x7f62800703c0 2023-12-25T21:06:43.278 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.274+0000 7f626effd700 1 -- 172.21.15.61:0/3759814433 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f627400f070 con 0x7f62800703c0 2023-12-25T21:06:43.278 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.274+0000 7f6284e18700 1 -- 172.21.15.61:0/3759814433 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6280045ad0 con 0x7f62800703c0 2023-12-25T21:06:43.278 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.274+0000 7f626effd700 1 --2- 172.21.15.61:0/3759814433 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f626402fd70 0x7f6264032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:43.278 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.274+0000 7f626effd700 1 -- 172.21.15.61:0/3759814433 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f62740413d0 con 0x7f62800703c0 2023-12-25T21:06:43.279 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.274+0000 7f627dd9b700 1 --2- 172.21.15.61:0/3759814433 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f626402fd70 0x7f6264032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:43.279 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.275+0000 7f627dd9b700 1 --2- 172.21.15.61:0/3759814433 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f626402fd70 0x7f6264032230 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f62700096a0 tx=0x7f6270006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:43.279 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.278+0000 7f626effd700 1 -- 172.21.15.61:0/3759814433 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62740224f0 con 0x7f62800703c0 2023-12-25T21:06:43.438 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.438+0000 7f626effd700 1 -- 172.21.15.61:0/3759814433 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f6274022d80 con 0x7f62800703c0 2023-12-25T21:06:43.465 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.464+0000 7f6284e18700 1 -- 172.21.15.61:0/3759814433 --> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f628010be10 con 0x7f626402fd70 2023-12-25T21:06:43.468 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.468+0000 7f626effd700 1 -- 172.21.15.61:0/3759814433 <== mgr.14144 v2:172.21.15.61:6800/2156288800 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f628010be10 con 0x7f626402fd70 2023-12-25T21:06:43.468 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-12-25T21:06:43.471 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.470+0000 7f6284e18700 1 -- 172.21.15.61:0/3759814433 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f626402fd70 msgr2=0x7f6264032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:43.471 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.470+0000 7f6284e18700 1 --2- 172.21.15.61:0/3759814433 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f626402fd70 0x7f6264032230 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f62700096a0 tx=0x7f6270006b40).stop 2023-12-25T21:06:43.471 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.470+0000 7f6284e18700 1 -- 172.21.15.61:0/3759814433 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62800703c0 msgr2=0x7f628010a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:43.472 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.470+0000 7f6284e18700 1 --2- 172.21.15.61:0/3759814433 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62800703c0 0x7f628010a9d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f627400e040 tx=0x7f6274006ee0).stop 2023-12-25T21:06:43.472 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.470+0000 7f6284e18700 1 -- 172.21.15.61:0/3759814433 shutdown_connections 2023-12-25T21:06:43.472 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.470+0000 7f6284e18700 1 --2- 172.21.15.61:0/3759814433 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f626402fd70 0x7f6264032230 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:43.472 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.470+0000 7f6284e18700 1 --2- 172.21.15.61:0/3759814433 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62800703c0 0x7f628010a9d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:43.472 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.470+0000 7f6284e18700 1 -- 172.21.15.61:0/3759814433 >> 172.21.15.61:0/3759814433 conn(0x7f62800f63b0 msgr2=0x7f62800f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:43.472 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.470+0000 7f6284e18700 1 -- 172.21.15.61:0/3759814433 shutdown_connections 2023-12-25T21:06:43.472 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:43.470+0000 7f6284e18700 1 -- 172.21.15.61:0/3759814433 wait complete. 2023-12-25T21:06:43.779 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:43 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:42.435975+0000 mgr.smithi061.jxxeke (mgr.14144) 3 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T21:06:43.780 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:43 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cephadm 2023-12-25T21:06:42.436458+0000 mgr.smithi061.jxxeke (mgr.14144) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2023-12-25T21:06:43.780 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:43 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:43.440145+0000 mon.smithi061 (mon.0) 80 : cluster [DBG] mgrmap e12: smithi061.jxxeke(active, since 2s) 2023-12-25T21:06:43.780 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:43 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:43.468884+0000 mon.smithi061 (mon.0) 81 : audit [INF] from='mgr.14144 172.21.15.61:0/408974420' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:43.835 INFO:teuthology.orchestra.run.smithi061.stderr:Deploying alertmanager service with default placement... 2023-12-25T21:06:43.836 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2023-12-25T21:06:44.226 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.221+0000 7f467e43c700 1 Processor -- start 2023-12-25T21:06:44.226 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.221+0000 7f467e43c700 1 -- start start 2023-12-25T21:06:44.226 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.221+0000 7f467e43c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46780fab80 0x7f46780fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:44.226 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.221+0000 7f467e43c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46780fb490 con 0x7f46780fab80 2023-12-25T21:06:44.226 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.221+0000 7f4677fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46780fab80 0x7f46780fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:44.226 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.221+0000 7f4677fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46780fab80 0x7f46780fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:36394/0 (socket says 172.21.15.61:36394) 2023-12-25T21:06:44.226 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.221+0000 7f4677fff700 1 -- 172.21.15.61:0/2980778783 learned_addr learned my addr 172.21.15.61:0/2980778783 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:44.227 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.221+0000 7f4677fff700 1 -- 172.21.15.61:0/2980778783 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46780fb5d0 con 0x7f46780fab80 2023-12-25T21:06:44.227 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.221+0000 7f4677fff700 1 --2- 172.21.15.61:0/2980778783 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46780fab80 0x7f46780fafa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f466800b5f0 tx=0x7f4668006ee0).ready entity=mon.0 client_cookie=20813de9080ac4fe server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:44.227 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.221+0000 7f4676ffd700 1 -- 172.21.15.61:0/2980778783 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f466800d600 con 0x7f46780fab80 2023-12-25T21:06:44.227 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.222+0000 7f4676ffd700 1 -- 172.21.15.61:0/2980778783 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f466800dbd0 con 0x7f46780fab80 2023-12-25T21:06:44.227 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.222+0000 7f4676ffd700 1 -- 172.21.15.61:0/2980778783 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4668011c00 con 0x7f46780fab80 2023-12-25T21:06:44.227 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.222+0000 7f467e43c700 1 -- 172.21.15.61:0/2980778783 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46780fab80 msgr2=0x7f46780fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:44.227 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.222+0000 7f467e43c700 1 --2- 172.21.15.61:0/2980778783 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46780fab80 0x7f46780fafa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f466800b5f0 tx=0x7f4668006ee0).stop 2023-12-25T21:06:44.228 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.222+0000 7f467e43c700 1 -- 172.21.15.61:0/2980778783 shutdown_connections 2023-12-25T21:06:44.228 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.222+0000 7f467e43c700 1 --2- 172.21.15.61:0/2980778783 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46780fab80 0x7f46780fafa0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:44.228 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.222+0000 7f467e43c700 1 -- 172.21.15.61:0/2980778783 >> 172.21.15.61:0/2980778783 conn(0x7f46780f63c0 msgr2=0x7f46780f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:44.228 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.222+0000 7f467e43c700 1 -- 172.21.15.61:0/2980778783 shutdown_connections 2023-12-25T21:06:44.228 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.222+0000 7f467e43c700 1 -- 172.21.15.61:0/2980778783 wait complete. 2023-12-25T21:06:44.228 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.223+0000 7f467e43c700 1 Processor -- start 2023-12-25T21:06:44.228 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.223+0000 7f467e43c700 1 -- start start 2023-12-25T21:06:44.228 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.223+0000 7f467e43c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46780fab80 0x7f467810fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:44.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.223+0000 7f467e43c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467810b530 con 0x7f46780fab80 2023-12-25T21:06:44.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.223+0000 7f4677fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46780fab80 0x7f467810fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:44.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.223+0000 7f4677fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46780fab80 0x7f467810fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:36410/0 (socket says 172.21.15.61:36410) 2023-12-25T21:06:44.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.223+0000 7f4677fff700 1 -- 172.21.15.61:0/3074330676 learned_addr learned my addr 172.21.15.61:0/3074330676 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:44.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.224+0000 7f4677fff700 1 -- 172.21.15.61:0/3074330676 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466800b200 con 0x7f46780fab80 2023-12-25T21:06:44.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.224+0000 7f4677fff700 1 --2- 172.21.15.61:0/3074330676 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46780fab80 0x7f467810fb40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f466800e040 tx=0x7f4668006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:44.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.224+0000 7f4674ff9700 1 -- 172.21.15.61:0/3074330676 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f466801b410 con 0x7f46780fab80 2023-12-25T21:06:44.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.224+0000 7f4674ff9700 1 -- 172.21.15.61:0/3074330676 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f466801b9e0 con 0x7f46780fab80 2023-12-25T21:06:44.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.224+0000 7f4674ff9700 1 -- 172.21.15.61:0/3074330676 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f466801a820 con 0x7f46780fab80 2023-12-25T21:06:44.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.224+0000 7f467e43c700 1 -- 172.21.15.61:0/3074330676 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4678070450 con 0x7f46780fab80 2023-12-25T21:06:44.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.224+0000 7f467e43c700 1 -- 172.21.15.61:0/3074330676 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4678110030 con 0x7f46780fab80 2023-12-25T21:06:44.229 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.225+0000 7f4674ff9700 1 -- 172.21.15.61:0/3074330676 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f466800f070 con 0x7f46780fab80 2023-12-25T21:06:44.230 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.225+0000 7f4674ff9700 1 --2- 172.21.15.61:0/3074330676 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f466402feb0 0x7f4664032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:44.230 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.225+0000 7f4674ff9700 1 -- 172.21.15.61:0/3074330676 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f4668041510 con 0x7f46780fab80 2023-12-25T21:06:44.230 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.225+0000 7f467e43c700 1 -- 172.21.15.61:0/3074330676 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4678045ad0 con 0x7f46780fab80 2023-12-25T21:06:44.230 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.225+0000 7f46777fe700 1 --2- 172.21.15.61:0/3074330676 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f466402feb0 0x7f4664032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:44.230 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.225+0000 7f46777fe700 1 --2- 172.21.15.61:0/3074330676 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f466402feb0 0x7f4664032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f466c0096a0 tx=0x7f466c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:44.230 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.229+0000 7f4674ff9700 1 -- 172.21.15.61:0/3074330676 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4668016070 con 0x7f46780fab80 2023-12-25T21:06:44.416 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.416+0000 7f467e43c700 1 -- 172.21.15.61:0/3074330676 --> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f46780fc960 con 0x7f466402feb0 2023-12-25T21:06:44.420 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.419+0000 7f4674ff9700 1 -- 172.21.15.61:0/3074330676 <== mgr.14144 v2:172.21.15.61:6800/2156288800 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f46780fc960 con 0x7f466402feb0 2023-12-25T21:06:44.420 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-12-25T21:06:44.422 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.421+0000 7f467e43c700 1 -- 172.21.15.61:0/3074330676 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f466402feb0 msgr2=0x7f4664032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:44.422 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.421+0000 7f467e43c700 1 --2- 172.21.15.61:0/3074330676 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f466402feb0 0x7f4664032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f466c0096a0 tx=0x7f466c006b40).stop 2023-12-25T21:06:44.422 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.421+0000 7f467e43c700 1 -- 172.21.15.61:0/3074330676 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46780fab80 msgr2=0x7f467810fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:44.422 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.421+0000 7f467e43c700 1 --2- 172.21.15.61:0/3074330676 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46780fab80 0x7f467810fb40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f466800e040 tx=0x7f4668006ee0).stop 2023-12-25T21:06:44.423 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.421+0000 7f467e43c700 1 -- 172.21.15.61:0/3074330676 shutdown_connections 2023-12-25T21:06:44.423 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.421+0000 7f467e43c700 1 --2- 172.21.15.61:0/3074330676 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f466402feb0 0x7f4664032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:44.423 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.421+0000 7f467e43c700 1 --2- 172.21.15.61:0/3074330676 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46780fab80 0x7f467810fb40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:44.423 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.421+0000 7f467e43c700 1 -- 172.21.15.61:0/3074330676 >> 172.21.15.61:0/3074330676 conn(0x7f46780f63c0 msgr2=0x7f46780f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:44.423 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.421+0000 7f467e43c700 1 -- 172.21.15.61:0/3074330676 shutdown_connections 2023-12-25T21:06:44.423 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:44.421+0000 7f467e43c700 1 -- 172.21.15.61:0/3074330676 wait complete. 2023-12-25T21:06:44.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:44 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:43.466363+0000 mgr.smithi061.jxxeke (mgr.14144) 5 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T21:06:44.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:44 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cephadm 2023-12-25T21:06:43.466999+0000 mgr.smithi061.jxxeke (mgr.14144) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2023-12-25T21:06:44.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:44 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:44.420209+0000 mon.smithi061 (mon.0) 82 : audit [INF] from='mgr.14144 172.21.15.61:0/408974420' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:44.769 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2023-12-25T21:06:45.174 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.171+0000 7f47f5dbf700 1 Processor -- start 2023-12-25T21:06:45.174 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.171+0000 7f47f5dbf700 1 -- start start 2023-12-25T21:06:45.174 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.172+0000 7f47f5dbf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47f00fab90 0x7f47f00fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:45.174 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.172+0000 7f47f5dbf700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f00fb4a0 con 0x7f47f00fab90 2023-12-25T21:06:45.174 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.172+0000 7f47ef7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47f00fab90 0x7f47f00fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:45.174 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.172+0000 7f47ef7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47f00fab90 0x7f47f00fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:36426/0 (socket says 172.21.15.61:36426) 2023-12-25T21:06:45.174 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.172+0000 7f47ef7fe700 1 -- 172.21.15.61:0/2030758655 learned_addr learned my addr 172.21.15.61:0/2030758655 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:45.175 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.172+0000 7f47ef7fe700 1 -- 172.21.15.61:0/2030758655 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f00fb5e0 con 0x7f47f00fab90 2023-12-25T21:06:45.175 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.172+0000 7f47ef7fe700 1 --2- 172.21.15.61:0/2030758655 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47f00fab90 0x7f47f00fafb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f47e000b5f0 tx=0x7f47e0006ee0).ready entity=mon.0 client_cookie=a528b7ad13c5559c server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:45.175 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.172+0000 7f47ee7fc700 1 -- 172.21.15.61:0/2030758655 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47e000d600 con 0x7f47f00fab90 2023-12-25T21:06:45.175 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.172+0000 7f47ee7fc700 1 -- 172.21.15.61:0/2030758655 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f47e000dbd0 con 0x7f47f00fab90 2023-12-25T21:06:45.175 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.172+0000 7f47ee7fc700 1 -- 172.21.15.61:0/2030758655 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47e0011c00 con 0x7f47f00fab90 2023-12-25T21:06:45.175 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.173+0000 7f47f5dbf700 1 -- 172.21.15.61:0/2030758655 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47f00fab90 msgr2=0x7f47f00fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:45.175 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.173+0000 7f47f5dbf700 1 --2- 172.21.15.61:0/2030758655 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47f00fab90 0x7f47f00fafb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f47e000b5f0 tx=0x7f47e0006ee0).stop 2023-12-25T21:06:45.175 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.173+0000 7f47f5dbf700 1 -- 172.21.15.61:0/2030758655 shutdown_connections 2023-12-25T21:06:45.175 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.173+0000 7f47f5dbf700 1 --2- 172.21.15.61:0/2030758655 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47f00fab90 0x7f47f00fafb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:45.175 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.173+0000 7f47f5dbf700 1 -- 172.21.15.61:0/2030758655 >> 172.21.15.61:0/2030758655 conn(0x7f47f00f63b0 msgr2=0x7f47f00f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:45.175 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.173+0000 7f47f5dbf700 1 -- 172.21.15.61:0/2030758655 shutdown_connections 2023-12-25T21:06:45.175 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.173+0000 7f47f5dbf700 1 -- 172.21.15.61:0/2030758655 wait complete. 2023-12-25T21:06:45.175 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.174+0000 7f47f5dbf700 1 Processor -- start 2023-12-25T21:06:45.177 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.174+0000 7f47f5dbf700 1 -- start start 2023-12-25T21:06:45.177 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.174+0000 7f47f5dbf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47f00fab90 0x7f47f0100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:45.177 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.174+0000 7f47f5dbf700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f00ff2a0 con 0x7f47f00fab90 2023-12-25T21:06:45.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.174+0000 7f47ef7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47f00fab90 0x7f47f0100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:45.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.174+0000 7f47ef7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47f00fab90 0x7f47f0100e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:36432/0 (socket says 172.21.15.61:36432) 2023-12-25T21:06:45.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.174+0000 7f47ef7fe700 1 -- 172.21.15.61:0/598286850 learned_addr learned my addr 172.21.15.61:0/598286850 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:45.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.175+0000 7f47ef7fe700 1 -- 172.21.15.61:0/598286850 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e000b200 con 0x7f47f00fab90 2023-12-25T21:06:45.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.175+0000 7f47ef7fe700 1 --2- 172.21.15.61:0/598286850 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47f00fab90 0x7f47f0100e20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f47e000e040 tx=0x7f47e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:45.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.175+0000 7f47e7fff700 1 -- 172.21.15.61:0/598286850 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47e001b410 con 0x7f47f00fab90 2023-12-25T21:06:45.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.175+0000 7f47e7fff700 1 -- 172.21.15.61:0/598286850 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f47e001b9e0 con 0x7f47f00fab90 2023-12-25T21:06:45.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.175+0000 7f47f5dbf700 1 -- 172.21.15.61:0/598286850 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f00ff440 con 0x7f47f00fab90 2023-12-25T21:06:45.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.175+0000 7f47e7fff700 1 -- 172.21.15.61:0/598286850 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47e001a820 con 0x7f47f00fab90 2023-12-25T21:06:45.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.175+0000 7f47f5dbf700 1 -- 172.21.15.61:0/598286850 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f00ff850 con 0x7f47f00fab90 2023-12-25T21:06:45.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.176+0000 7f47e7fff700 1 -- 172.21.15.61:0/598286850 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f47e000f070 con 0x7f47f00fab90 2023-12-25T21:06:45.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.176+0000 7f47e7fff700 1 --2- 172.21.15.61:0/598286850 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f47d002feb0 0x7f47d0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:45.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.176+0000 7f47e7fff700 1 -- 172.21.15.61:0/598286850 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f47e001bb50 con 0x7f47f00fab90 2023-12-25T21:06:45.178 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.176+0000 7f47eeffd700 1 --2- 172.21.15.61:0/598286850 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f47d002feb0 0x7f47d0032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:45.179 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.176+0000 7f47f5dbf700 1 -- 172.21.15.61:0/598286850 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f0059100 con 0x7f47f00fab90 2023-12-25T21:06:45.179 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.176+0000 7f47eeffd700 1 --2- 172.21.15.61:0/598286850 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f47d002feb0 0x7f47d0032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f47d80096a0 tx=0x7f47d8006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:45.181 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.180+0000 7f47e7fff700 1 -- 172.21.15.61:0/598286850 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47e0016070 con 0x7f47f00fab90 2023-12-25T21:06:45.339 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.338+0000 7f47f5dbf700 1 -- 172.21.15.61:0/598286850 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f47f0104940 con 0x7f47f00fab90 2023-12-25T21:06:45.342 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.341+0000 7f47e7fff700 1 -- 172.21.15.61:0/598286850 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f47e0025460 con 0x7f47f00fab90 2023-12-25T21:06:45.345 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.343+0000 7f47f5dbf700 1 -- 172.21.15.61:0/598286850 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f47d002feb0 msgr2=0x7f47d0032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:45.346 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.343+0000 7f47f5dbf700 1 --2- 172.21.15.61:0/598286850 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f47d002feb0 0x7f47d0032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f47d80096a0 tx=0x7f47d8006b40).stop 2023-12-25T21:06:45.346 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.343+0000 7f47f5dbf700 1 -- 172.21.15.61:0/598286850 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47f00fab90 msgr2=0x7f47f0100e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:45.346 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.343+0000 7f47f5dbf700 1 --2- 172.21.15.61:0/598286850 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47f00fab90 0x7f47f0100e20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f47e000e040 tx=0x7f47e0006ee0).stop 2023-12-25T21:06:45.346 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.343+0000 7f47f5dbf700 1 -- 172.21.15.61:0/598286850 shutdown_connections 2023-12-25T21:06:45.346 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.343+0000 7f47f5dbf700 1 --2- 172.21.15.61:0/598286850 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f47d002feb0 0x7f47d0032370 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:45.347 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.343+0000 7f47f5dbf700 1 --2- 172.21.15.61:0/598286850 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47f00fab90 0x7f47f0100e20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:45.347 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.343+0000 7f47f5dbf700 1 -- 172.21.15.61:0/598286850 >> 172.21.15.61:0/598286850 conn(0x7f47f00f63b0 msgr2=0x7f47f00f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:45.347 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.344+0000 7f47f5dbf700 1 -- 172.21.15.61:0/598286850 shutdown_connections 2023-12-25T21:06:45.347 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:45.344+0000 7f47f5dbf700 1 -- 172.21.15.61:0/598286850 wait complete. 2023-12-25T21:06:45.671 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:45 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:44.417546+0000 mgr.smithi061.jxxeke (mgr.14144) 7 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T21:06:45.671 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:45 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cephadm 2023-12-25T21:06:44.418064+0000 mgr.smithi061.jxxeke (mgr.14144) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-12-25T21:06:45.671 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:45 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:45.342468+0000 mon.smithi061 (mon.0) 83 : audit [INF] from='client.? 172.21.15.61:0/598286850' entity='client.admin' 2023-12-25T21:06:45.702 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2023-12-25T21:06:46.079 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.077+0000 7f2bc73a6700 1 Processor -- start 2023-12-25T21:06:46.079 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.078+0000 7f2bc73a6700 1 -- start start 2023-12-25T21:06:46.079 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.078+0000 7f2bc73a6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc0070390 0x7f2bc0100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:46.079 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.078+0000 7f2bc73a6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc00fc770 con 0x7f2bc0070390 2023-12-25T21:06:46.079 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.078+0000 7f2bc5142700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc0070390 0x7f2bc0100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:46.079 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.078+0000 7f2bc5142700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc0070390 0x7f2bc0100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:36440/0 (socket says 172.21.15.61:36440) 2023-12-25T21:06:46.083 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.078+0000 7f2bc5142700 1 -- 172.21.15.61:0/3443401155 learned_addr learned my addr 172.21.15.61:0/3443401155 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:46.083 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.078+0000 7f2bc5142700 1 -- 172.21.15.61:0/3443401155 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc0101270 con 0x7f2bc0070390 2023-12-25T21:06:46.083 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.078+0000 7f2bc5142700 1 --2- 172.21.15.61:0/3443401155 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc0070390 0x7f2bc0100d80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f2bb000b5f0 tx=0x7f2bb0006ee0).ready entity=mon.0 client_cookie=9a1e17cd72e37388 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:46.083 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.079+0000 7f2baffff700 1 -- 172.21.15.61:0/3443401155 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb000d600 con 0x7f2bc0070390 2023-12-25T21:06:46.083 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.079+0000 7f2baffff700 1 -- 172.21.15.61:0/3443401155 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2bb000dbd0 con 0x7f2bc0070390 2023-12-25T21:06:46.083 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.079+0000 7f2baffff700 1 -- 172.21.15.61:0/3443401155 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb0011c00 con 0x7f2bc0070390 2023-12-25T21:06:46.083 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.079+0000 7f2bc73a6700 1 -- 172.21.15.61:0/3443401155 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc0070390 msgr2=0x7f2bc0100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:46.083 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.079+0000 7f2bc73a6700 1 --2- 172.21.15.61:0/3443401155 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc0070390 0x7f2bc0100d80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f2bb000b5f0 tx=0x7f2bb0006ee0).stop 2023-12-25T21:06:46.083 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.079+0000 7f2bc73a6700 1 -- 172.21.15.61:0/3443401155 shutdown_connections 2023-12-25T21:06:46.083 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.079+0000 7f2bc73a6700 1 --2- 172.21.15.61:0/3443401155 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc0070390 0x7f2bc0100d80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:46.084 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.079+0000 7f2bc73a6700 1 -- 172.21.15.61:0/3443401155 >> 172.21.15.61:0/3443401155 conn(0x7f2bc00f63c0 msgr2=0x7f2bc00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:46.084 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.079+0000 7f2bc73a6700 1 -- 172.21.15.61:0/3443401155 shutdown_connections 2023-12-25T21:06:46.084 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.079+0000 7f2bc73a6700 1 -- 172.21.15.61:0/3443401155 wait complete. 2023-12-25T21:06:46.084 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.080+0000 7f2bc73a6700 1 Processor -- start 2023-12-25T21:06:46.084 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.080+0000 7f2bc73a6700 1 -- start start 2023-12-25T21:06:46.084 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.080+0000 7f2bc73a6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc0070390 0x7f2bc010cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:46.084 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.080+0000 7f2bc73a6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc010d200 con 0x7f2bc0070390 2023-12-25T21:06:46.084 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.080+0000 7f2bc5142700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc0070390 0x7f2bc010cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:46.084 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.080+0000 7f2bc5142700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc0070390 0x7f2bc010cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:36448/0 (socket says 172.21.15.61:36448) 2023-12-25T21:06:46.084 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.080+0000 7f2bc5142700 1 -- 172.21.15.61:0/3515200141 learned_addr learned my addr 172.21.15.61:0/3515200141 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:46.084 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.080+0000 7f2bc5142700 1 -- 172.21.15.61:0/3515200141 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb000b200 con 0x7f2bc0070390 2023-12-25T21:06:46.084 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.080+0000 7f2bc5142700 1 --2- 172.21.15.61:0/3515200141 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc0070390 0x7f2bc010cd10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f2bb000e040 tx=0x7f2bb0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:46.084 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.081+0000 7f2badffb700 1 -- 172.21.15.61:0/3515200141 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb000d9f0 con 0x7f2bc0070390 2023-12-25T21:06:46.084 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.081+0000 7f2badffb700 1 -- 172.21.15.61:0/3515200141 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2bb001a9f0 con 0x7f2bc0070390 2023-12-25T21:06:46.085 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.081+0000 7f2bc73a6700 1 -- 172.21.15.61:0/3515200141 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc010d400 con 0x7f2bc0070390 2023-12-25T21:06:46.085 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.081+0000 7f2badffb700 1 -- 172.21.15.61:0/3515200141 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb001b8c0 con 0x7f2bc0070390 2023-12-25T21:06:46.085 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.081+0000 7f2bc73a6700 1 -- 172.21.15.61:0/3515200141 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc010d810 con 0x7f2bc0070390 2023-12-25T21:06:46.085 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.082+0000 7f2badffb700 1 -- 172.21.15.61:0/3515200141 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f2bb000f070 con 0x7f2bc0070390 2023-12-25T21:06:46.085 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.082+0000 7f2badffb700 1 --2- 172.21.15.61:0/3515200141 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f2ba802fe60 0x7f2ba8032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:46.085 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.082+0000 7f2badffb700 1 -- 172.21.15.61:0/3515200141 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f2bb0041570 con 0x7f2bc0070390 2023-12-25T21:06:46.085 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.082+0000 7f2bc73a6700 1 -- 172.21.15.61:0/3515200141 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc0045ad0 con 0x7f2bc0070390 2023-12-25T21:06:46.085 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.082+0000 7f2bc4941700 1 --2- 172.21.15.61:0/3515200141 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f2ba802fe60 0x7f2ba8032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:46.085 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.082+0000 7f2bc4941700 1 --2- 172.21.15.61:0/3515200141 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f2ba802fe60 0x7f2ba8032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f2bbc0096a0 tx=0x7f2bbc006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:46.087 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.086+0000 7f2badffb700 1 -- 172.21.15.61:0/3515200141 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2bb00224f0 con 0x7f2bc0070390 2023-12-25T21:06:46.244 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.244+0000 7f2bc73a6700 1 -- 172.21.15.61:0/3515200141 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f2bc0101d70 con 0x7f2bc0070390 2023-12-25T21:06:46.247 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.247+0000 7f2badffb700 1 -- 172.21.15.61:0/3515200141 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7f2bb0006a70 con 0x7f2bc0070390 2023-12-25T21:06:46.250 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.249+0000 7f2bc73a6700 1 -- 172.21.15.61:0/3515200141 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f2ba802fe60 msgr2=0x7f2ba8032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:46.250 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.249+0000 7f2bc73a6700 1 --2- 172.21.15.61:0/3515200141 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f2ba802fe60 0x7f2ba8032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f2bbc0096a0 tx=0x7f2bbc006b40).stop 2023-12-25T21:06:46.250 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.249+0000 7f2bc73a6700 1 -- 172.21.15.61:0/3515200141 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc0070390 msgr2=0x7f2bc010cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:46.250 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.249+0000 7f2bc73a6700 1 --2- 172.21.15.61:0/3515200141 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc0070390 0x7f2bc010cd10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f2bb000e040 tx=0x7f2bb0006ee0).stop 2023-12-25T21:06:46.250 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.249+0000 7f2bc73a6700 1 -- 172.21.15.61:0/3515200141 shutdown_connections 2023-12-25T21:06:46.250 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.249+0000 7f2bc73a6700 1 --2- 172.21.15.61:0/3515200141 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f2ba802fe60 0x7f2ba8032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:46.250 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.249+0000 7f2bc73a6700 1 --2- 172.21.15.61:0/3515200141 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc0070390 0x7f2bc010cd10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:46.251 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.249+0000 7f2bc73a6700 1 -- 172.21.15.61:0/3515200141 >> 172.21.15.61:0/3515200141 conn(0x7f2bc00f63c0 msgr2=0x7f2bc00f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:46.251 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.249+0000 7f2bc73a6700 1 -- 172.21.15.61:0/3515200141 shutdown_connections 2023-12-25T21:06:46.251 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.249+0000 7f2bc73a6700 1 -- 172.21.15.61:0/3515200141 wait complete. 2023-12-25T21:06:46.593 INFO:teuthology.orchestra.run.smithi061.stderr:Enabling the dashboard module... 2023-12-25T21:06:46.594 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2023-12-25T21:06:47.000 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.998+0000 7fc87212c700 1 Processor -- start 2023-12-25T21:06:47.000 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.998+0000 7fc87212c700 1 -- start start 2023-12-25T21:06:47.000 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.998+0000 7fc87212c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc86c0703f0 0x7fc86c0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:47.000 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.998+0000 7fc87212c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c0fee10 con 0x7fc86c0703f0 2023-12-25T21:06:47.000 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.999+0000 7fc86b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc86c0703f0 0x7fc86c0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:47.000 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.999+0000 7fc86b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc86c0703f0 0x7fc86c0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:36458/0 (socket says 172.21.15.61:36458) 2023-12-25T21:06:47.000 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.999+0000 7fc86b7fe700 1 -- 172.21.15.61:0/3290401486 learned_addr learned my addr 172.21.15.61:0/3290401486 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:47.000 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.999+0000 7fc86b7fe700 1 -- 172.21.15.61:0/3290401486 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c0fef50 con 0x7fc86c0703f0 2023-12-25T21:06:47.003 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.999+0000 7fc86b7fe700 1 --2- 172.21.15.61:0/3290401486 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc86c0703f0 0x7fc86c0fe920 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fc86000b5f0 tx=0x7fc860006ee0).ready entity=mon.0 client_cookie=b634ec87c058bf67 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:47.003 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.999+0000 7fc86a7fc700 1 -- 172.21.15.61:0/3290401486 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc86000d600 con 0x7fc86c0703f0 2023-12-25T21:06:47.003 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.999+0000 7fc86a7fc700 1 -- 172.21.15.61:0/3290401486 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc86000dbd0 con 0x7fc86c0703f0 2023-12-25T21:06:47.003 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:46.999+0000 7fc86a7fc700 1 -- 172.21.15.61:0/3290401486 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc860011c00 con 0x7fc86c0703f0 2023-12-25T21:06:47.004 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.000+0000 7fc87212c700 1 -- 172.21.15.61:0/3290401486 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc86c0703f0 msgr2=0x7fc86c0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:47.004 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.000+0000 7fc87212c700 1 --2- 172.21.15.61:0/3290401486 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc86c0703f0 0x7fc86c0fe920 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fc86000b5f0 tx=0x7fc860006ee0).stop 2023-12-25T21:06:47.004 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.000+0000 7fc87212c700 1 -- 172.21.15.61:0/3290401486 shutdown_connections 2023-12-25T21:06:47.004 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.000+0000 7fc87212c700 1 --2- 172.21.15.61:0/3290401486 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc86c0703f0 0x7fc86c0fe920 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:47.004 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.000+0000 7fc87212c700 1 -- 172.21.15.61:0/3290401486 >> 172.21.15.61:0/3290401486 conn(0x7fc86c0f63c0 msgr2=0x7fc86c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:47.004 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.000+0000 7fc87212c700 1 -- 172.21.15.61:0/3290401486 shutdown_connections 2023-12-25T21:06:47.004 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.000+0000 7fc87212c700 1 -- 172.21.15.61:0/3290401486 wait complete. 2023-12-25T21:06:47.004 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.000+0000 7fc87212c700 1 Processor -- start 2023-12-25T21:06:47.004 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.001+0000 7fc87212c700 1 -- start start 2023-12-25T21:06:47.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.001+0000 7fc87212c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc86c0703f0 0x7fc86c0fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:47.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.001+0000 7fc87212c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c0fa820 con 0x7fc86c0703f0 2023-12-25T21:06:47.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.001+0000 7fc86b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc86c0703f0 0x7fc86c0fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:47.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.001+0000 7fc86b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc86c0703f0 0x7fc86c0fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:36462/0 (socket says 172.21.15.61:36462) 2023-12-25T21:06:47.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.001+0000 7fc86b7fe700 1 -- 172.21.15.61:0/4155817949 learned_addr learned my addr 172.21.15.61:0/4155817949 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:47.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.001+0000 7fc86b7fe700 1 -- 172.21.15.61:0/4155817949 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86000b200 con 0x7fc86c0703f0 2023-12-25T21:06:47.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.001+0000 7fc86b7fe700 1 --2- 172.21.15.61:0/4155817949 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc86c0703f0 0x7fc86c0fc3a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fc86000b190 tx=0x7fc860006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:47.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.001+0000 7fc853fff700 1 -- 172.21.15.61:0/4155817949 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc86001b410 con 0x7fc86c0703f0 2023-12-25T21:06:47.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.001+0000 7fc853fff700 1 -- 172.21.15.61:0/4155817949 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc86001b9e0 con 0x7fc86c0703f0 2023-12-25T21:06:47.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.001+0000 7fc853fff700 1 -- 172.21.15.61:0/4155817949 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc86001a880 con 0x7fc86c0703f0 2023-12-25T21:06:47.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.001+0000 7fc87212c700 1 -- 172.21.15.61:0/4155817949 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc86c0fa9c0 con 0x7fc86c0703f0 2023-12-25T21:06:47.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.002+0000 7fc87212c700 1 -- 172.21.15.61:0/4155817949 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc86c0fadd0 con 0x7fc86c0703f0 2023-12-25T21:06:47.005 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.002+0000 7fc853fff700 1 -- 172.21.15.61:0/4155817949 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fc86000f070 con 0x7fc86c0703f0 2023-12-25T21:06:47.006 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.002+0000 7fc853fff700 1 --2- 172.21.15.61:0/4155817949 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7fc85802feb0 0x7fc858032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:47.006 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.002+0000 7fc853fff700 1 -- 172.21.15.61:0/4155817949 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fc860041350 con 0x7fc86c0703f0 2023-12-25T21:06:47.006 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.003+0000 7fc86affd700 1 --2- 172.21.15.61:0/4155817949 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7fc85802feb0 0x7fc858032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:47.006 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.003+0000 7fc87212c700 1 -- 172.21.15.61:0/4155817949 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc86c059100 con 0x7fc86c0703f0 2023-12-25T21:06:47.006 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.003+0000 7fc86affd700 1 --2- 172.21.15.61:0/4155817949 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7fc85802feb0 0x7fc858032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fc85c0096a0 tx=0x7fc85c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:47.008 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.008+0000 7fc853fff700 1 -- 172.21.15.61:0/4155817949 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc860025460 con 0x7fc86c0703f0 2023-12-25T21:06:47.207 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.206+0000 7fc87212c700 1 -- 172.21.15.61:0/4155817949 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fc86c045ad0 con 0x7fc86c0703f0 2023-12-25T21:06:47.275 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.275+0000 7fc853fff700 1 -- 172.21.15.61:0/4155817949 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fc8600257e0 con 0x7fc86c0703f0 2023-12-25T21:06:47.277 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.276+0000 7fc853fff700 1 -- 172.21.15.61:0/4155817949 <== mon.0 v2:172.21.15.61:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7fc8600239b0 con 0x7fc86c0703f0 2023-12-25T21:06:47.279 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.278+0000 7fc87212c700 1 -- 172.21.15.61:0/4155817949 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7fc85802feb0 msgr2=0x7fc858032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:47.279 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.278+0000 7fc87212c700 1 --2- 172.21.15.61:0/4155817949 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7fc85802feb0 0x7fc858032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fc85c0096a0 tx=0x7fc85c006b40).stop 2023-12-25T21:06:47.279 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.278+0000 7fc87212c700 1 -- 172.21.15.61:0/4155817949 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc86c0703f0 msgr2=0x7fc86c0fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:47.279 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.278+0000 7fc87212c700 1 --2- 172.21.15.61:0/4155817949 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc86c0703f0 0x7fc86c0fc3a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fc86000b190 tx=0x7fc860006ee0).stop 2023-12-25T21:06:47.280 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.278+0000 7fc87212c700 1 -- 172.21.15.61:0/4155817949 shutdown_connections 2023-12-25T21:06:47.280 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.278+0000 7fc87212c700 1 --2- 172.21.15.61:0/4155817949 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7fc85802feb0 0x7fc858032370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:47.280 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.278+0000 7fc87212c700 1 --2- 172.21.15.61:0/4155817949 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc86c0703f0 0x7fc86c0fc3a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:47.280 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.278+0000 7fc87212c700 1 -- 172.21.15.61:0/4155817949 >> 172.21.15.61:0/4155817949 conn(0x7fc86c0f63c0 msgr2=0x7fc86c0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:47.280 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.279+0000 7fc87212c700 1 -- 172.21.15.61:0/4155817949 shutdown_connections 2023-12-25T21:06:47.280 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:47.279+0000 7fc87212c700 1 -- 172.21.15.61:0/4155817949 wait complete. 2023-12-25T21:06:47.498 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:47 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:46.247954+0000 mon.smithi061 (mon.0) 84 : audit [INF] from='client.? 172.21.15.61:0/3515200141' entity='client.admin' 2023-12-25T21:06:47.498 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:47 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:47.207707+0000 mon.smithi061 (mon.0) 85 : audit [INF] from='client.? 172.21.15.61:0/4155817949' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-12-25T21:06:47.719 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-12-25T21:06:48.250 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.249+0000 7f534f8ca700 1 Processor -- start 2023-12-25T21:06:48.250 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.249+0000 7f534f8ca700 1 -- start start 2023-12-25T21:06:48.250 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.249+0000 7f534f8ca700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5348070430 0x7f53480fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:48.251 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.249+0000 7f534f8ca700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53480fedd0 con 0x7f5348070430 2023-12-25T21:06:48.252 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.250+0000 7f534d666700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5348070430 0x7f53480fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:48.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.250+0000 7f534d666700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5348070430 0x7f53480fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:36482/0 (socket says 172.21.15.61:36482) 2023-12-25T21:06:48.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.250+0000 7f534d666700 1 -- 172.21.15.61:0/1597071605 learned_addr learned my addr 172.21.15.61:0/1597071605 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:48.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.250+0000 7f534d666700 1 -- 172.21.15.61:0/1597071605 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53480fef10 con 0x7f5348070430 2023-12-25T21:06:48.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.250+0000 7f534d666700 1 --2- 172.21.15.61:0/1597071605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5348070430 0x7f53480fe8e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f534400b5f0 tx=0x7f5344006ee0).ready entity=mon.0 client_cookie=6618f820ca320bdd server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:48.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.250+0000 7f5337fff700 1 -- 172.21.15.61:0/1597071605 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f534400d600 con 0x7f5348070430 2023-12-25T21:06:48.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.250+0000 7f5337fff700 1 -- 172.21.15.61:0/1597071605 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f534400dbd0 con 0x7f5348070430 2023-12-25T21:06:48.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.251+0000 7f5337fff700 1 -- 172.21.15.61:0/1597071605 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5344011c00 con 0x7f5348070430 2023-12-25T21:06:48.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.251+0000 7f534f8ca700 1 -- 172.21.15.61:0/1597071605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5348070430 msgr2=0x7f53480fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:48.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.251+0000 7f534f8ca700 1 --2- 172.21.15.61:0/1597071605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5348070430 0x7f53480fe8e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f534400b5f0 tx=0x7f5344006ee0).stop 2023-12-25T21:06:48.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.251+0000 7f534f8ca700 1 -- 172.21.15.61:0/1597071605 shutdown_connections 2023-12-25T21:06:48.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.251+0000 7f534f8ca700 1 --2- 172.21.15.61:0/1597071605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5348070430 0x7f53480fe8e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:48.253 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.251+0000 7f534f8ca700 1 -- 172.21.15.61:0/1597071605 >> 172.21.15.61:0/1597071605 conn(0x7f53480f63c0 msgr2=0x7f53480f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:48.254 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.251+0000 7f534f8ca700 1 -- 172.21.15.61:0/1597071605 shutdown_connections 2023-12-25T21:06:48.254 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.251+0000 7f534f8ca700 1 -- 172.21.15.61:0/1597071605 wait complete. 2023-12-25T21:06:48.255 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.252+0000 7f534f8ca700 1 Processor -- start 2023-12-25T21:06:48.255 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.252+0000 7f534f8ca700 1 -- start start 2023-12-25T21:06:48.255 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.252+0000 7f534f8ca700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5348070430 0x7f53480fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:48.255 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.253+0000 7f534f8ca700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53480fa7e0 con 0x7f5348070430 2023-12-25T21:06:48.255 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.253+0000 7f534d666700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5348070430 0x7f53480fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:48.255 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.253+0000 7f534d666700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5348070430 0x7f53480fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:36488/0 (socket says 172.21.15.61:36488) 2023-12-25T21:06:48.255 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.253+0000 7f534d666700 1 -- 172.21.15.61:0/1903824206 learned_addr learned my addr 172.21.15.61:0/1903824206 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:48.255 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.253+0000 7f534d666700 1 -- 172.21.15.61:0/1903824206 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534400b200 con 0x7f5348070430 2023-12-25T21:06:48.255 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.253+0000 7f534d666700 1 --2- 172.21.15.61:0/1903824206 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5348070430 0x7f53480fc360 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f534400e040 tx=0x7f5344006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:48.256 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.253+0000 7f5335ffb700 1 -- 172.21.15.61:0/1903824206 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f534400d9f0 con 0x7f5348070430 2023-12-25T21:06:48.256 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.253+0000 7f534f8ca700 1 -- 172.21.15.61:0/1903824206 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53480fa980 con 0x7f5348070430 2023-12-25T21:06:48.256 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.253+0000 7f5335ffb700 1 -- 172.21.15.61:0/1903824206 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f534401a9f0 con 0x7f5348070430 2023-12-25T21:06:48.256 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.253+0000 7f5335ffb700 1 -- 172.21.15.61:0/1903824206 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f534401b8c0 con 0x7f5348070430 2023-12-25T21:06:48.256 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.254+0000 7f534f8ca700 1 -- 172.21.15.61:0/1903824206 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53480fad90 con 0x7f5348070430 2023-12-25T21:06:48.256 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.255+0000 7f5335ffb700 1 -- 172.21.15.61:0/1903824206 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f534400f070 con 0x7f5348070430 2023-12-25T21:06:48.257 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.255+0000 7f534f8ca700 1 -- 172.21.15.61:0/1903824206 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5348045ad0 con 0x7f5348070430 2023-12-25T21:06:48.257 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.255+0000 7f5335ffb700 1 --2- 172.21.15.61:0/1903824206 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f533002ff00 0x7f53300323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:48.257 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.255+0000 7f5335ffb700 1 -- 172.21.15.61:0/1903824206 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f5344041450 con 0x7f5348070430 2023-12-25T21:06:48.257 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.255+0000 7f534ce65700 1 -- 172.21.15.61:0/1903824206 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f533002ff00 msgr2=0x7f53300323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2156288800 2023-12-25T21:06:48.257 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.255+0000 7f534ce65700 1 --2- 172.21.15.61:0/1903824206 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f533002ff00 0x7f53300323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-25T21:06:48.260 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.259+0000 7f5335ffb700 1 -- 172.21.15.61:0/1903824206 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5344022d70 con 0x7f5348070430 2023-12-25T21:06:48.457 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.456+0000 7f534ce65700 1 -- 172.21.15.61:0/1903824206 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f533002ff00 msgr2=0x7f53300323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2156288800 2023-12-25T21:06:48.457 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.456+0000 7f534ce65700 1 --2- 172.21.15.61:0/1903824206 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f533002ff00 0x7f53300323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-25T21:06:48.460 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.460+0000 7f534f8ca700 1 -- 172.21.15.61:0/1903824206 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f53480ffca0 con 0x7f5348070430 2023-12-25T21:06:48.461 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.460+0000 7f5335ffb700 1 -- 172.21.15.61:0/1903824206 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f5344016070 con 0x7f5348070430 2023-12-25T21:06:48.461 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: { 2023-12-25T21:06:48.461 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "epoch": 13, 2023-12-25T21:06:48.461 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "available": true, 2023-12-25T21:06:48.461 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "active_name": "smithi061.jxxeke", 2023-12-25T21:06:48.462 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "num_standby": 0 2023-12-25T21:06:48.462 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: } 2023-12-25T21:06:48.463 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.462+0000 7f534f8ca700 1 -- 172.21.15.61:0/1903824206 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f533002ff00 msgr2=0x7f53300323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-25T21:06:48.463 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.462+0000 7f534f8ca700 1 --2- 172.21.15.61:0/1903824206 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f533002ff00 0x7f53300323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T21:06:48.463 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.462+0000 7f534f8ca700 1 -- 172.21.15.61:0/1903824206 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5348070430 msgr2=0x7f53480fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:48.463 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.462+0000 7f534f8ca700 1 --2- 172.21.15.61:0/1903824206 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5348070430 0x7f53480fc360 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f534400e040 tx=0x7f5344006ee0).stop 2023-12-25T21:06:48.463 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.462+0000 7f534f8ca700 1 -- 172.21.15.61:0/1903824206 shutdown_connections 2023-12-25T21:06:48.463 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.462+0000 7f534f8ca700 1 --2- 172.21.15.61:0/1903824206 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f533002ff00 0x7f53300323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T21:06:48.464 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.462+0000 7f534f8ca700 1 --2- 172.21.15.61:0/1903824206 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5348070430 0x7f53480fc360 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:48.464 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.462+0000 7f534f8ca700 1 -- 172.21.15.61:0/1903824206 >> 172.21.15.61:0/1903824206 conn(0x7f53480f63c0 msgr2=0x7f53480f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:48.464 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.462+0000 7f534f8ca700 1 -- 172.21.15.61:0/1903824206 shutdown_connections 2023-12-25T21:06:48.464 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:48.462+0000 7f534f8ca700 1 -- 172.21.15.61:0/1903824206 wait complete. 2023-12-25T21:06:48.577 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:48 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:47.277457+0000 mon.smithi061 (mon.0) 86 : audit [INF] from='client.? 172.21.15.61:0/4155817949' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-12-25T21:06:48.578 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:48 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:47.277502+0000 mon.smithi061 (mon.0) 87 : cluster [DBG] mgrmap e13: smithi061.jxxeke(active, since 6s) 2023-12-25T21:06:49.085 INFO:teuthology.orchestra.run.smithi061.stderr:Waiting for the mgr to restart... 2023-12-25T21:06:49.085 INFO:teuthology.orchestra.run.smithi061.stderr:Waiting for mgr epoch 13... 2023-12-25T21:06:49.085 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-12-25T21:06:49.455 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:49 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:48.461296+0000 mon.smithi061 (mon.0) 88 : audit [DBG] from='client.? 172.21.15.61:0/1903824206' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-25T21:06:49.460 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.459+0000 7f6c4d2ea700 1 Processor -- start 2023-12-25T21:06:49.461 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.459+0000 7f6c4d2ea700 1 -- start start 2023-12-25T21:06:49.461 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.459+0000 7f6c4d2ea700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c480fb150 0x7f6c480fb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:49.461 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.459+0000 7f6c4d2ea700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c480fba60 con 0x7f6c480fb150 2023-12-25T21:06:49.461 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.460+0000 7f6c46ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c480fb150 0x7f6c480fb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:49.461 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.460+0000 7f6c46ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c480fb150 0x7f6c480fb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:40742/0 (socket says 172.21.15.61:40742) 2023-12-25T21:06:49.461 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.460+0000 7f6c46ffd700 1 -- 172.21.15.61:0/3560063148 learned_addr learned my addr 172.21.15.61:0/3560063148 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:49.463 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.460+0000 7f6c46ffd700 1 -- 172.21.15.61:0/3560063148 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c480fbba0 con 0x7f6c480fb150 2023-12-25T21:06:49.463 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.460+0000 7f6c46ffd700 1 --2- 172.21.15.61:0/3560063148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c480fb150 0x7f6c480fb570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f6c3800b5f0 tx=0x7f6c38006ee0).ready entity=mon.0 client_cookie=c5241155f14f5f91 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:49.463 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.460+0000 7f6c45ffb700 1 -- 172.21.15.61:0/3560063148 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c3800d600 con 0x7f6c480fb150 2023-12-25T21:06:49.463 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.460+0000 7f6c45ffb700 1 -- 172.21.15.61:0/3560063148 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6c3800dbd0 con 0x7f6c480fb150 2023-12-25T21:06:49.463 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.460+0000 7f6c45ffb700 1 -- 172.21.15.61:0/3560063148 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c38011c00 con 0x7f6c480fb150 2023-12-25T21:06:49.463 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.461+0000 7f6c4d2ea700 1 -- 172.21.15.61:0/3560063148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c480fb150 msgr2=0x7f6c480fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:49.463 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.461+0000 7f6c4d2ea700 1 --2- 172.21.15.61:0/3560063148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c480fb150 0x7f6c480fb570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f6c3800b5f0 tx=0x7f6c38006ee0).stop 2023-12-25T21:06:49.463 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.461+0000 7f6c4d2ea700 1 -- 172.21.15.61:0/3560063148 shutdown_connections 2023-12-25T21:06:49.463 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.461+0000 7f6c4d2ea700 1 --2- 172.21.15.61:0/3560063148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c480fb150 0x7f6c480fb570 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:49.463 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.461+0000 7f6c4d2ea700 1 -- 172.21.15.61:0/3560063148 >> 172.21.15.61:0/3560063148 conn(0x7f6c480f5b40 msgr2=0x7f6c480f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:49.463 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.461+0000 7f6c4d2ea700 1 -- 172.21.15.61:0/3560063148 shutdown_connections 2023-12-25T21:06:49.464 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.461+0000 7f6c4d2ea700 1 -- 172.21.15.61:0/3560063148 wait complete. 2023-12-25T21:06:49.466 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.462+0000 7f6c4d2ea700 1 Processor -- start 2023-12-25T21:06:49.466 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.462+0000 7f6c4d2ea700 1 -- start start 2023-12-25T21:06:49.466 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.462+0000 7f6c4d2ea700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c480fb150 0x7f6c4810ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:49.467 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.462+0000 7f6c4d2ea700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4810d1b0 con 0x7f6c480fb150 2023-12-25T21:06:49.467 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.462+0000 7f6c46ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c480fb150 0x7f6c4810ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:49.467 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.462+0000 7f6c46ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c480fb150 0x7f6c4810ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:40752/0 (socket says 172.21.15.61:40752) 2023-12-25T21:06:49.467 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.462+0000 7f6c46ffd700 1 -- 172.21.15.61:0/956859294 learned_addr learned my addr 172.21.15.61:0/956859294 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:49.467 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.462+0000 7f6c46ffd700 1 -- 172.21.15.61:0/956859294 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3800b200 con 0x7f6c480fb150 2023-12-25T21:06:49.468 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.462+0000 7f6c46ffd700 1 --2- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c480fb150 0x7f6c4810ccc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f6c3800e040 tx=0x7f6c38006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:49.468 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.462+0000 7f6c2f7fe700 1 -- 172.21.15.61:0/956859294 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c3801b410 con 0x7f6c480fb150 2023-12-25T21:06:49.468 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.463+0000 7f6c2f7fe700 1 -- 172.21.15.61:0/956859294 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6c3801b9e0 con 0x7f6c480fb150 2023-12-25T21:06:49.468 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.463+0000 7f6c4d2ea700 1 -- 172.21.15.61:0/956859294 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c4810d3b0 con 0x7f6c480fb150 2023-12-25T21:06:49.468 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.463+0000 7f6c2f7fe700 1 -- 172.21.15.61:0/956859294 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c3801a820 con 0x7f6c480fb150 2023-12-25T21:06:49.468 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.463+0000 7f6c4d2ea700 1 -- 172.21.15.61:0/956859294 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c4810d7c0 con 0x7f6c480fb150 2023-12-25T21:06:49.468 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.463+0000 7f6c2f7fe700 1 -- 172.21.15.61:0/956859294 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f6c3800f070 con 0x7f6c480fb150 2023-12-25T21:06:49.469 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.463+0000 7f6c2f7fe700 1 --2- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f6c3402ff00 0x7f6c340323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:49.469 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.463+0000 7f6c4d2ea700 1 -- 172.21.15.61:0/956859294 --> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c48059100 con 0x7f6c3402ff00 2023-12-25T21:06:49.469 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.464+0000 7f6c2f7fe700 1 -- 172.21.15.61:0/956859294 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f6c38041530 con 0x7f6c480fb150 2023-12-25T21:06:49.470 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.464+0000 7f6c467fc700 1 -- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f6c3402ff00 msgr2=0x7f6c340323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2156288800 2023-12-25T21:06:49.470 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.464+0000 7f6c467fc700 1 --2- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f6c3402ff00 0x7f6c340323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-25T21:06:49.665 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.665+0000 7f6c467fc700 1 -- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f6c3402ff00 msgr2=0x7f6c340323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2156288800 2023-12-25T21:06:49.666 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:49.665+0000 7f6c467fc700 1 --2- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f6c3402ff00 0x7f6c340323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-25T21:06:50.066 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:50.066+0000 7f6c467fc700 1 -- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f6c3402ff00 msgr2=0x7f6c340323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2156288800 2023-12-25T21:06:50.067 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:50.066+0000 7f6c467fc700 1 --2- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f6c3402ff00 0x7f6c340323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-25T21:06:50.867 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:50.867+0000 7f6c467fc700 1 -- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f6c3402ff00 msgr2=0x7f6c340323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2156288800 2023-12-25T21:06:50.868 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:50.867+0000 7f6c467fc700 1 --2- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f6c3402ff00 0x7f6c340323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-12-25T21:06:52.470 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:52.468+0000 7f6c467fc700 1 -- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f6c3402ff00 msgr2=0x7f6c340323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.61:6800/2156288800 2023-12-25T21:06:52.470 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:52.468+0000 7f6c467fc700 1 --2- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f6c3402ff00 0x7f6c340323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-12-25T21:06:52.561 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:52.560+0000 7f6c2f7fe700 1 -- 172.21.15.61:0/956859294 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f6c380257e0 con 0x7f6c480fb150 2023-12-25T21:06:52.561 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:52.560+0000 7f6c2f7fe700 1 -- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f6c3402ff00 msgr2=0x7f6c340323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-25T21:06:52.561 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:52.560+0000 7f6c2f7fe700 1 --2- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f6c3402ff00 0x7f6c340323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T21:06:52.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:52 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:52.508915+0000 mon.smithi061 (mon.0) 89 : cluster [INF] Active manager daemon smithi061.jxxeke restarted 2023-12-25T21:06:52.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:52 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:52.509313+0000 mon.smithi061 (mon.0) 90 : cluster [INF] Activating manager daemon smithi061.jxxeke 2023-12-25T21:06:52.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:52 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:52.511652+0000 mon.smithi061 (mon.0) 91 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-12-25T21:06:53.565 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:53.564+0000 7f6c2f7fe700 1 -- 172.21.15.61:0/956859294 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f6c380224f0 con 0x7f6c480fb150 2023-12-25T21:06:53.565 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:53.564+0000 7f6c2f7fe700 1 --2- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6c340339e0 0x7f6c34035dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:53.565 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:53.564+0000 7f6c2f7fe700 1 -- 172.21.15.61:0/956859294 --> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c48059100 con 0x7f6c340339e0 2023-12-25T21:06:53.565 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:53.564+0000 7f6c467fc700 1 --2- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6c340339e0 0x7f6c34035dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:53.566 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:53.565+0000 7f6c467fc700 1 --2- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6c340339e0 0x7f6c34035dd0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f6c3c006fd0 tx=0x7f6c3c006d70).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:53.566 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:53.565+0000 7f6c2f7fe700 1 -- 172.21.15.61:0/956859294 <== mgr.14164 v2:172.21.15.61:6800/889227411 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f6c48059100 con 0x7f6c340339e0 2023-12-25T21:06:53.573 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:53.573+0000 7f6c4d2ea700 1 -- 172.21.15.61:0/956859294 --> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f6c48045ad0 con 0x7f6c340339e0 2023-12-25T21:06:53.574 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:53.573+0000 7f6c2f7fe700 1 -- 172.21.15.61:0/956859294 <== mgr.14164 v2:172.21.15.61:6800/889227411 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f6c48045ad0 con 0x7f6c340339e0 2023-12-25T21:06:53.574 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: { 2023-12-25T21:06:53.574 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2023-12-25T21:06:53.574 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: "initialized": true 2023-12-25T21:06:53.574 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: } 2023-12-25T21:06:53.576 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:53.573+0000 7f6c4d2ea700 1 -- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6c340339e0 msgr2=0x7f6c34035dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:53.576 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:53.573+0000 7f6c4d2ea700 1 --2- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6c340339e0 0x7f6c34035dd0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f6c3c006fd0 tx=0x7f6c3c006d70).stop 2023-12-25T21:06:53.576 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:53.573+0000 7f6c4d2ea700 1 -- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c480fb150 msgr2=0x7f6c4810ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:53.576 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:53.573+0000 7f6c4d2ea700 1 --2- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c480fb150 0x7f6c4810ccc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f6c3800e040 tx=0x7f6c38006ee0).stop 2023-12-25T21:06:53.576 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:53.574+0000 7f6c4d2ea700 1 -- 172.21.15.61:0/956859294 shutdown_connections 2023-12-25T21:06:53.576 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:53.574+0000 7f6c4d2ea700 1 --2- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6c340339e0 0x7f6c34035dd0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:53.576 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:53.574+0000 7f6c4d2ea700 1 --2- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c480fb150 0x7f6c4810ccc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:53.576 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:53.574+0000 7f6c4d2ea700 1 --2- 172.21.15.61:0/956859294 >> [v2:172.21.15.61:6800/2156288800,v1:172.21.15.61:6801/2156288800] conn(0x7f6c3402ff00 0x7f6c340323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-25T21:06:53.576 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:53.574+0000 7f6c4d2ea700 1 -- 172.21.15.61:0/956859294 >> 172.21.15.61:0/956859294 conn(0x7f6c480f5b40 msgr2=0x7f6c480fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:53.576 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:53.574+0000 7f6c4d2ea700 1 -- 172.21.15.61:0/956859294 shutdown_connections 2023-12-25T21:06:53.576 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:53.574+0000 7f6c4d2ea700 1 -- 172.21.15.61:0/956859294 wait complete. 2023-12-25T21:06:53.931 INFO:teuthology.orchestra.run.smithi061.stderr:mgr epoch 13 is available 2023-12-25T21:06:53.931 INFO:teuthology.orchestra.run.smithi061.stderr:Generating a dashboard self-signed certificate... 2023-12-25T21:06:53.931 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2023-12-25T21:06:53.933 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:52.562029+0000 mon.smithi061 (mon.0) 92 : cluster [DBG] mgrmap e14: smithi061.jxxeke(active, starting, since 0.0527979s) 2023-12-25T21:06:53.934 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:52.564688+0000 mon.smithi061 (mon.0) 93 : audit [DBG] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "mon metadata", "id": "smithi061"}]: dispatch 2023-12-25T21:06:53.934 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:52.565349+0000 mon.smithi061 (mon.0) 94 : audit [DBG] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "mgr metadata", "who": "smithi061.jxxeke", "id": "smithi061.jxxeke"}]: dispatch 2023-12-25T21:06:53.934 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:52.566036+0000 mon.smithi061 (mon.0) 95 : audit [DBG] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-25T21:06:53.934 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:52.566210+0000 mon.smithi061 (mon.0) 96 : audit [DBG] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-25T21:06:53.934 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:52.566349+0000 mon.smithi061 (mon.0) 97 : audit [DBG] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-25T21:06:53.934 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:52.568282+0000 mon.smithi061 (mon.0) 98 : cluster [INF] Manager daemon smithi061.jxxeke is now available 2023-12-25T21:06:53.934 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:52.673240+0000 mon.smithi061 (mon.0) 99 : audit [DBG] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-25T21:06:53.934 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:52.675618+0000 mon.smithi061 (mon.0) 100 : audit [DBG] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-25T21:06:53.934 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:52.699142+0000 mon.smithi061 (mon.0) 101 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/mirror_snapshot_schedule"}]: dispatch 2023-12-25T21:06:53.934 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:52.702614+0000 mon.smithi061 (mon.0) 102 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/trash_purge_schedule"}]: dispatch 2023-12-25T21:06:54.296 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.294+0000 7f25d4efd700 1 Processor -- start 2023-12-25T21:06:54.296 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.294+0000 7f25d4efd700 1 -- start start 2023-12-25T21:06:54.296 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.294+0000 7f25d4efd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25d0070430 0x7f25d00fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:54.296 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.294+0000 7f25d4efd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d00fec20 con 0x7f25d0070430 2023-12-25T21:06:54.296 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.294+0000 7f25cf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25d0070430 0x7f25d00fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:54.296 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.294+0000 7f25cf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25d0070430 0x7f25d00fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:40808/0 (socket says 172.21.15.61:40808) 2023-12-25T21:06:54.296 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.294+0000 7f25cf7fe700 1 -- 172.21.15.61:0/3693753724 learned_addr learned my addr 172.21.15.61:0/3693753724 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:54.297 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.295+0000 7f25cf7fe700 1 -- 172.21.15.61:0/3693753724 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d00fed60 con 0x7f25d0070430 2023-12-25T21:06:54.297 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.295+0000 7f25cf7fe700 1 --2- 172.21.15.61:0/3693753724 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25d0070430 0x7f25d00fe730 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f25c000b5f0 tx=0x7f25c0006ee0).ready entity=mon.0 client_cookie=72ed38bb9c9c16d4 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:54.297 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.295+0000 7f25ce7fc700 1 -- 172.21.15.61:0/3693753724 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25c000d600 con 0x7f25d0070430 2023-12-25T21:06:54.297 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.295+0000 7f25ce7fc700 1 -- 172.21.15.61:0/3693753724 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f25c000dbd0 con 0x7f25d0070430 2023-12-25T21:06:54.297 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.295+0000 7f25ce7fc700 1 -- 172.21.15.61:0/3693753724 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25c0011c00 con 0x7f25d0070430 2023-12-25T21:06:54.297 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.295+0000 7f25d4efd700 1 -- 172.21.15.61:0/3693753724 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25d0070430 msgr2=0x7f25d00fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:54.297 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.295+0000 7f25d4efd700 1 --2- 172.21.15.61:0/3693753724 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25d0070430 0x7f25d00fe730 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f25c000b5f0 tx=0x7f25c0006ee0).stop 2023-12-25T21:06:54.298 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.295+0000 7f25d4efd700 1 -- 172.21.15.61:0/3693753724 shutdown_connections 2023-12-25T21:06:54.298 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.295+0000 7f25d4efd700 1 --2- 172.21.15.61:0/3693753724 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25d0070430 0x7f25d00fe730 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:54.298 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.295+0000 7f25d4efd700 1 -- 172.21.15.61:0/3693753724 >> 172.21.15.61:0/3693753724 conn(0x7f25d00f6210 msgr2=0x7f25d00f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:54.298 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.296+0000 7f25d4efd700 1 -- 172.21.15.61:0/3693753724 shutdown_connections 2023-12-25T21:06:54.298 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.296+0000 7f25d4efd700 1 -- 172.21.15.61:0/3693753724 wait complete. 2023-12-25T21:06:54.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.296+0000 7f25d4efd700 1 Processor -- start 2023-12-25T21:06:54.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.296+0000 7f25d4efd700 1 -- start start 2023-12-25T21:06:54.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.296+0000 7f25d4efd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25d0070430 0x7f25d010a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:54.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.296+0000 7f25d4efd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d010ad20 con 0x7f25d0070430 2023-12-25T21:06:54.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.297+0000 7f25cf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25d0070430 0x7f25d010a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:54.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.297+0000 7f25cf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25d0070430 0x7f25d010a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:40820/0 (socket says 172.21.15.61:40820) 2023-12-25T21:06:54.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.297+0000 7f25cf7fe700 1 -- 172.21.15.61:0/378108774 learned_addr learned my addr 172.21.15.61:0/378108774 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:54.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.297+0000 7f25cf7fe700 1 -- 172.21.15.61:0/378108774 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c000b200 con 0x7f25d0070430 2023-12-25T21:06:54.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.297+0000 7f25cf7fe700 1 --2- 172.21.15.61:0/378108774 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25d0070430 0x7f25d010a830 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f25c000b190 tx=0x7f25c0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:54.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.297+0000 7f25b7fff700 1 -- 172.21.15.61:0/378108774 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25c001b410 con 0x7f25d0070430 2023-12-25T21:06:54.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.297+0000 7f25b7fff700 1 -- 172.21.15.61:0/378108774 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f25c001b9e0 con 0x7f25d0070430 2023-12-25T21:06:54.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.297+0000 7f25d4efd700 1 -- 172.21.15.61:0/378108774 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d010af20 con 0x7f25d0070430 2023-12-25T21:06:54.300 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.297+0000 7f25b7fff700 1 -- 172.21.15.61:0/378108774 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25c001a880 con 0x7f25d0070430 2023-12-25T21:06:54.301 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.297+0000 7f25d4efd700 1 -- 172.21.15.61:0/378108774 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d010b330 con 0x7f25d0070430 2023-12-25T21:06:54.301 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.298+0000 7f25b7fff700 1 -- 172.21.15.61:0/378108774 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f25c000f070 con 0x7f25d0070430 2023-12-25T21:06:54.301 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.298+0000 7f25b7fff700 1 --2- 172.21.15.61:0/378108774 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f25bc02fd50 0x7f25bc032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:54.301 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.298+0000 7f25b7fff700 1 -- 172.21.15.61:0/378108774 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25c0041420 con 0x7f25d0070430 2023-12-25T21:06:54.301 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.298+0000 7f25ceffd700 1 --2- 172.21.15.61:0/378108774 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f25bc02fd50 0x7f25bc032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:54.301 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.298+0000 7f25d4efd700 1 -- 172.21.15.61:0/378108774 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d0059100 con 0x7f25d0070430 2023-12-25T21:06:54.301 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.299+0000 7f25ceffd700 1 --2- 172.21.15.61:0/378108774 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f25bc02fd50 0x7f25bc032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f25c40096a0 tx=0x7f25c4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:54.303 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.303+0000 7f25b7fff700 1 -- 172.21.15.61:0/378108774 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25c0016070 con 0x7f25d0070430 2023-12-25T21:06:54.466 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.466+0000 7f25d4efd700 1 -- 172.21.15.61:0/378108774 --> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f25d01046b0 con 0x7f25bc02fd50 2023-12-25T21:06:54.570 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.570+0000 7f25b7fff700 1 -- 172.21.15.61:0/378108774 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f25c0025980 con 0x7f25d0070430 2023-12-25T21:06:54.681 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.680+0000 7f25b7fff700 1 -- 172.21.15.61:0/378108774 <== mgr.14164 v2:172.21.15.61:6800/889227411 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f25d01046b0 con 0x7f25bc02fd50 2023-12-25T21:06:54.681 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: Self-signed certificate created 2023-12-25T21:06:54.683 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.682+0000 7f25d4efd700 1 -- 172.21.15.61:0/378108774 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f25bc02fd50 msgr2=0x7f25bc032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:54.683 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.682+0000 7f25d4efd700 1 --2- 172.21.15.61:0/378108774 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f25bc02fd50 0x7f25bc032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f25c40096a0 tx=0x7f25c4006b40).stop 2023-12-25T21:06:54.684 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.682+0000 7f25d4efd700 1 -- 172.21.15.61:0/378108774 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25d0070430 msgr2=0x7f25d010a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:54.684 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.682+0000 7f25d4efd700 1 --2- 172.21.15.61:0/378108774 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25d0070430 0x7f25d010a830 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f25c000b190 tx=0x7f25c0006ee0).stop 2023-12-25T21:06:54.684 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.682+0000 7f25d4efd700 1 -- 172.21.15.61:0/378108774 shutdown_connections 2023-12-25T21:06:54.684 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.682+0000 7f25d4efd700 1 --2- 172.21.15.61:0/378108774 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f25bc02fd50 0x7f25bc032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:54.684 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.682+0000 7f25d4efd700 1 --2- 172.21.15.61:0/378108774 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25d0070430 0x7f25d010a830 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:54.684 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.682+0000 7f25d4efd700 1 -- 172.21.15.61:0/378108774 >> 172.21.15.61:0/378108774 conn(0x7f25d00f6210 msgr2=0x7f25d00f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:54.684 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.682+0000 7f25d4efd700 1 -- 172.21.15.61:0/378108774 shutdown_connections 2023-12-25T21:06:54.684 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:54.682+0000 7f25d4efd700 1 -- 172.21.15.61:0/378108774 wait complete. 2023-12-25T21:06:54.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:54 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:53.566581+0000 mgr.smithi061.jxxeke (mgr.14164) 1 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-25T21:06:54.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:54 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:53.566804+0000 mon.smithi061 (mon.0) 103 : cluster [DBG] mgrmap e15: smithi061.jxxeke(active, since 1.05757s) 2023-12-25T21:06:54.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:54 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:53.574336+0000 mgr.smithi061.jxxeke (mgr.14164) 2 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-25T21:06:55.142 INFO:teuthology.orchestra.run.smithi061.stderr:Creating initial admin user... 2023-12-25T21:06:55.143 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpclt3eb70:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-12-25T21:06:55.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.551+0000 7f1b90a33700 1 Processor -- start 2023-12-25T21:06:55.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.551+0000 7f1b90a33700 1 -- start start 2023-12-25T21:06:55.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.551+0000 7f1b90a33700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b8c070430 0x7f1b8c0fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:55.552 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.551+0000 7f1b90a33700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c0fec20 con 0x7f1b8c070430 2023-12-25T21:06:55.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.551+0000 7f1b8b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b8c070430 0x7f1b8c0fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:55.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.551+0000 7f1b8b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b8c070430 0x7f1b8c0fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:40836/0 (socket says 172.21.15.61:40836) 2023-12-25T21:06:55.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.551+0000 7f1b8b7fe700 1 -- 172.21.15.61:0/1788670514 learned_addr learned my addr 172.21.15.61:0/1788670514 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:55.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.552+0000 7f1b8b7fe700 1 -- 172.21.15.61:0/1788670514 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8c0fed60 con 0x7f1b8c070430 2023-12-25T21:06:55.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.552+0000 7f1b8b7fe700 1 --2- 172.21.15.61:0/1788670514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b8c070430 0x7f1b8c0fe730 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f1b7c00b5f0 tx=0x7f1b7c006ee0).ready entity=mon.0 client_cookie=ad4162bd8a70422f server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:55.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.552+0000 7f1b8a7fc700 1 -- 172.21.15.61:0/1788670514 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b7c00d600 con 0x7f1b8c070430 2023-12-25T21:06:55.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.552+0000 7f1b8a7fc700 1 -- 172.21.15.61:0/1788670514 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1b7c00dbd0 con 0x7f1b8c070430 2023-12-25T21:06:55.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.552+0000 7f1b8a7fc700 1 -- 172.21.15.61:0/1788670514 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b7c011c00 con 0x7f1b8c070430 2023-12-25T21:06:55.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.552+0000 7f1b90a33700 1 -- 172.21.15.61:0/1788670514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b8c070430 msgr2=0x7f1b8c0fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:55.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.552+0000 7f1b90a33700 1 --2- 172.21.15.61:0/1788670514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b8c070430 0x7f1b8c0fe730 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f1b7c00b5f0 tx=0x7f1b7c006ee0).stop 2023-12-25T21:06:55.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.553+0000 7f1b90a33700 1 -- 172.21.15.61:0/1788670514 shutdown_connections 2023-12-25T21:06:55.556 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.553+0000 7f1b90a33700 1 --2- 172.21.15.61:0/1788670514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b8c070430 0x7f1b8c0fe730 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:55.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.553+0000 7f1b90a33700 1 -- 172.21.15.61:0/1788670514 >> 172.21.15.61:0/1788670514 conn(0x7f1b8c0f6210 msgr2=0x7f1b8c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:55.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.553+0000 7f1b90a33700 1 -- 172.21.15.61:0/1788670514 shutdown_connections 2023-12-25T21:06:55.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.553+0000 7f1b90a33700 1 -- 172.21.15.61:0/1788670514 wait complete. 2023-12-25T21:06:55.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.553+0000 7f1b90a33700 1 Processor -- start 2023-12-25T21:06:55.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.553+0000 7f1b90a33700 1 -- start start 2023-12-25T21:06:55.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.554+0000 7f1b90a33700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b8c070430 0x7f1b8c10a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:55.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.554+0000 7f1b90a33700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c10ad40 con 0x7f1b8c070430 2023-12-25T21:06:55.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.554+0000 7f1b8b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b8c070430 0x7f1b8c10a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:55.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.554+0000 7f1b8b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b8c070430 0x7f1b8c10a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:40838/0 (socket says 172.21.15.61:40838) 2023-12-25T21:06:55.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.554+0000 7f1b8b7fe700 1 -- 172.21.15.61:0/2644148935 learned_addr learned my addr 172.21.15.61:0/2644148935 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:55.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.554+0000 7f1b8b7fe700 1 -- 172.21.15.61:0/2644148935 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7c00b200 con 0x7f1b8c070430 2023-12-25T21:06:55.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.554+0000 7f1b8b7fe700 1 --2- 172.21.15.61:0/2644148935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b8c070430 0x7f1b8c10a850 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f1b7c00b190 tx=0x7f1b7c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:55.557 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.554+0000 7f1b73fff700 1 -- 172.21.15.61:0/2644148935 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b7c01b410 con 0x7f1b8c070430 2023-12-25T21:06:55.558 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.554+0000 7f1b90a33700 1 -- 172.21.15.61:0/2644148935 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b8c10af40 con 0x7f1b8c070430 2023-12-25T21:06:55.558 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.554+0000 7f1b73fff700 1 -- 172.21.15.61:0/2644148935 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1b7c01b9e0 con 0x7f1b8c070430 2023-12-25T21:06:55.558 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.554+0000 7f1b73fff700 1 -- 172.21.15.61:0/2644148935 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b7c01a880 con 0x7f1b8c070430 2023-12-25T21:06:55.558 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.554+0000 7f1b90a33700 1 -- 172.21.15.61:0/2644148935 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8c10b350 con 0x7f1b8c070430 2023-12-25T21:06:55.558 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.555+0000 7f1b73fff700 1 -- 172.21.15.61:0/2644148935 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f1b7c00f070 con 0x7f1b8c070430 2023-12-25T21:06:55.558 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.555+0000 7f1b73fff700 1 --2- 172.21.15.61:0/2644148935 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1b7802feb0 0x7f1b78032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:55.559 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.555+0000 7f1b73fff700 1 -- 172.21.15.61:0/2644148935 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b7c041520 con 0x7f1b8c070430 2023-12-25T21:06:55.559 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.555+0000 7f1b8affd700 1 --2- 172.21.15.61:0/2644148935 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1b7802feb0 0x7f1b78032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:55.559 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.555+0000 7f1b90a33700 1 -- 172.21.15.61:0/2644148935 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8c059100 con 0x7f1b8c070430 2023-12-25T21:06:55.559 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.556+0000 7f1b8affd700 1 --2- 172.21.15.61:0/2644148935 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1b7802feb0 0x7f1b78032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f1b800096a0 tx=0x7f1b80006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:55.560 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.560+0000 7f1b73fff700 1 -- 172.21.15.61:0/2644148935 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b7c0224f0 con 0x7f1b8c070430 2023-12-25T21:06:55.734 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.734+0000 7f1b90a33700 1 -- 172.21.15.61:0/2644148935 --> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1b8c045ad0 con 0x7f1b7802feb0 2023-12-25T21:06:55.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:55 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:54.467560+0000 mgr.smithi061.jxxeke (mgr.14164) 3 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T21:06:55.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:55 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:54.571912+0000 mon.smithi061 (mon.0) 104 : cluster [DBG] mgrmap e16: smithi061.jxxeke(active, since 2s) 2023-12-25T21:06:55.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:55 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:54.678824+0000 mon.smithi061 (mon.0) 105 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:55.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:55 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:54.681341+0000 mon.smithi061 (mon.0) 106 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:55.981 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.980+0000 7f1b73fff700 1 -- 172.21.15.61:0/2644148935 <== mgr.14164 v2:172.21.15.61:6800/889227411 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f1b8c045ad0 con 0x7f1b7802feb0 2023-12-25T21:06:55.981 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$wxf2QY1299sufbcTXkmQtupshH7QWIVv4Nt/G5uA3n9z7VnHEoOMC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1703538415, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-12-25T21:06:55.983 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.982+0000 7f1b90a33700 1 -- 172.21.15.61:0/2644148935 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1b7802feb0 msgr2=0x7f1b78032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:55.983 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.982+0000 7f1b90a33700 1 --2- 172.21.15.61:0/2644148935 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1b7802feb0 0x7f1b78032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f1b800096a0 tx=0x7f1b80006b40).stop 2023-12-25T21:06:55.983 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.982+0000 7f1b90a33700 1 -- 172.21.15.61:0/2644148935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b8c070430 msgr2=0x7f1b8c10a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:55.983 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.982+0000 7f1b90a33700 1 --2- 172.21.15.61:0/2644148935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b8c070430 0x7f1b8c10a850 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f1b7c00b190 tx=0x7f1b7c006ee0).stop 2023-12-25T21:06:55.983 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.982+0000 7f1b90a33700 1 -- 172.21.15.61:0/2644148935 shutdown_connections 2023-12-25T21:06:55.983 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.982+0000 7f1b90a33700 1 --2- 172.21.15.61:0/2644148935 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1b7802feb0 0x7f1b78032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:55.983 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.982+0000 7f1b90a33700 1 --2- 172.21.15.61:0/2644148935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b8c070430 0x7f1b8c10a850 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:55.984 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.982+0000 7f1b90a33700 1 -- 172.21.15.61:0/2644148935 >> 172.21.15.61:0/2644148935 conn(0x7f1b8c0f6210 msgr2=0x7f1b8c0f7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:55.984 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.982+0000 7f1b90a33700 1 -- 172.21.15.61:0/2644148935 shutdown_connections 2023-12-25T21:06:55.984 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:55.982+0000 7f1b90a33700 1 -- 172.21.15.61:0/2644148935 wait complete. 2023-12-25T21:06:56.384 INFO:teuthology.orchestra.run.smithi061.stderr:Fetching dashboard port number... 2023-12-25T21:06:56.384 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmps7spmgc1:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpxu3_plo0:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2023-12-25T21:06:56.745 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.742+0000 7f7c2bfff700 1 Processor -- start 2023-12-25T21:06:56.745 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.742+0000 7f7c2bfff700 1 -- start start 2023-12-25T21:06:56.745 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.742+0000 7f7c2bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c2c0fa9f0 0x7f7c2c0fae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:56.745 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.742+0000 7f7c2bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c0fb300 con 0x7f7c2c0fa9f0 2023-12-25T21:06:56.745 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.742+0000 7f7c2affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c2c0fa9f0 0x7f7c2c0fae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:56.745 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.742+0000 7f7c2affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c2c0fa9f0 0x7f7c2c0fae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:40844/0 (socket says 172.21.15.61:40844) 2023-12-25T21:06:56.745 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.742+0000 7f7c2affd700 1 -- 172.21.15.61:0/2505899114 learned_addr learned my addr 172.21.15.61:0/2505899114 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:56.745 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.743+0000 7f7c2affd700 1 -- 172.21.15.61:0/2505899114 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2c0fb440 con 0x7f7c2c0fa9f0 2023-12-25T21:06:56.745 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.743+0000 7f7c2affd700 1 --2- 172.21.15.61:0/2505899114 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c2c0fa9f0 0x7f7c2c0fae10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f7c1c00b5f0 tx=0x7f7c1c006ee0).ready entity=mon.0 client_cookie=b0d6043622079c30 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:56.746 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.743+0000 7f7c29ffb700 1 -- 172.21.15.61:0/2505899114 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c1c00d600 con 0x7f7c2c0fa9f0 2023-12-25T21:06:56.746 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.743+0000 7f7c29ffb700 1 -- 172.21.15.61:0/2505899114 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7c1c00dbd0 con 0x7f7c2c0fa9f0 2023-12-25T21:06:56.746 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.743+0000 7f7c29ffb700 1 -- 172.21.15.61:0/2505899114 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c1c011c00 con 0x7f7c2c0fa9f0 2023-12-25T21:06:56.746 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.743+0000 7f7c2bfff700 1 -- 172.21.15.61:0/2505899114 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c2c0fa9f0 msgr2=0x7f7c2c0fae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:56.746 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.743+0000 7f7c2bfff700 1 --2- 172.21.15.61:0/2505899114 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c2c0fa9f0 0x7f7c2c0fae10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f7c1c00b5f0 tx=0x7f7c1c006ee0).stop 2023-12-25T21:06:56.746 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.743+0000 7f7c2bfff700 1 -- 172.21.15.61:0/2505899114 shutdown_connections 2023-12-25T21:06:56.746 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.743+0000 7f7c2bfff700 1 --2- 172.21.15.61:0/2505899114 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c2c0fa9f0 0x7f7c2c0fae10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:56.746 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.743+0000 7f7c2bfff700 1 -- 172.21.15.61:0/2505899114 >> 172.21.15.61:0/2505899114 conn(0x7f7c2c0f6250 msgr2=0x7f7c2c0f8670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:56.746 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.743+0000 7f7c2bfff700 1 -- 172.21.15.61:0/2505899114 shutdown_connections 2023-12-25T21:06:56.746 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.743+0000 7f7c2bfff700 1 -- 172.21.15.61:0/2505899114 wait complete. 2023-12-25T21:06:56.748 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.744+0000 7f7c2bfff700 1 Processor -- start 2023-12-25T21:06:56.749 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.744+0000 7f7c2bfff700 1 -- start start 2023-12-25T21:06:56.749 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.745+0000 7f7c2bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c2c0fa9f0 0x7f7c2c10d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:56.749 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.745+0000 7f7c2bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c0704c0 con 0x7f7c2c0fa9f0 2023-12-25T21:06:56.749 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.745+0000 7f7c2affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c2c0fa9f0 0x7f7c2c10d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:56.749 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.745+0000 7f7c2affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c2c0fa9f0 0x7f7c2c10d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:40858/0 (socket says 172.21.15.61:40858) 2023-12-25T21:06:56.749 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.745+0000 7f7c2affd700 1 -- 172.21.15.61:0/285574618 learned_addr learned my addr 172.21.15.61:0/285574618 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:56.749 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.745+0000 7f7c2affd700 1 -- 172.21.15.61:0/285574618 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c00b200 con 0x7f7c2c0fa9f0 2023-12-25T21:06:56.749 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.745+0000 7f7c2affd700 1 --2- 172.21.15.61:0/285574618 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c2c0fa9f0 0x7f7c2c10d5d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f7c1c00e040 tx=0x7f7c1c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:56.749 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.745+0000 7f7c137fe700 1 -- 172.21.15.61:0/285574618 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c1c01b410 con 0x7f7c2c0fa9f0 2023-12-25T21:06:56.749 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.745+0000 7f7c137fe700 1 -- 172.21.15.61:0/285574618 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7c1c01b9e0 con 0x7f7c2c0fa9f0 2023-12-25T21:06:56.749 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.745+0000 7f7c137fe700 1 -- 172.21.15.61:0/285574618 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c1c01a770 con 0x7f7c2c0fa9f0 2023-12-25T21:06:56.750 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.745+0000 7f7c2bfff700 1 -- 172.21.15.61:0/285574618 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2c0706c0 con 0x7f7c2c0fa9f0 2023-12-25T21:06:56.750 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.745+0000 7f7c2bfff700 1 -- 172.21.15.61:0/285574618 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c2c10dc80 con 0x7f7c2c0fa9f0 2023-12-25T21:06:56.750 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.746+0000 7f7c137fe700 1 -- 172.21.15.61:0/285574618 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f7c1c00f070 con 0x7f7c2c0fa9f0 2023-12-25T21:06:56.750 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.746+0000 7f7c137fe700 1 --2- 172.21.15.61:0/285574618 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7c1802feb0 0x7f7c18032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:56.750 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.746+0000 7f7c2bfff700 1 -- 172.21.15.61:0/285574618 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c2c045ad0 con 0x7f7c2c0fa9f0 2023-12-25T21:06:56.750 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.746+0000 7f7c137fe700 1 -- 172.21.15.61:0/285574618 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7c1c0412c0 con 0x7f7c2c0fa9f0 2023-12-25T21:06:56.750 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.747+0000 7f7c2a7fc700 1 --2- 172.21.15.61:0/285574618 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7c1802feb0 0x7f7c18032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:56.750 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.747+0000 7f7c2a7fc700 1 --2- 172.21.15.61:0/285574618 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7c1802feb0 0x7f7c18032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f7c200096a0 tx=0x7f7c20006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:56.751 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.751+0000 7f7c137fe700 1 -- 172.21.15.61:0/285574618 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c1c016070 con 0x7f7c2c0fa9f0 2023-12-25T21:06:56.908 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.907+0000 7f7c2bfff700 1 -- 172.21.15.61:0/285574618 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f7c2c0fc800 con 0x7f7c2c0fa9f0 2023-12-25T21:06:56.909 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.908+0000 7f7c137fe700 1 -- 172.21.15.61:0/285574618 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v9) v1 ==== 112+0+5 (secure 0 0 0) 0x7f7c1c024410 con 0x7f7c2c0fa9f0 2023-12-25T21:06:56.909 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 8443 2023-12-25T21:06:56.911 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.910+0000 7f7c2bfff700 1 -- 172.21.15.61:0/285574618 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7c1802feb0 msgr2=0x7f7c18032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:56.911 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.910+0000 7f7c2bfff700 1 --2- 172.21.15.61:0/285574618 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7c1802feb0 0x7f7c18032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f7c200096a0 tx=0x7f7c20006b40).stop 2023-12-25T21:06:56.911 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.910+0000 7f7c2bfff700 1 -- 172.21.15.61:0/285574618 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c2c0fa9f0 msgr2=0x7f7c2c10d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:56.911 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.910+0000 7f7c2bfff700 1 --2- 172.21.15.61:0/285574618 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c2c0fa9f0 0x7f7c2c10d5d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f7c1c00e040 tx=0x7f7c1c006ee0).stop 2023-12-25T21:06:56.911 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.910+0000 7f7c2bfff700 1 -- 172.21.15.61:0/285574618 shutdown_connections 2023-12-25T21:06:56.911 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.910+0000 7f7c2bfff700 1 --2- 172.21.15.61:0/285574618 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7c1802feb0 0x7f7c18032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:56.911 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.910+0000 7f7c2bfff700 1 --2- 172.21.15.61:0/285574618 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c2c0fa9f0 0x7f7c2c10d5d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:56.911 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.910+0000 7f7c2bfff700 1 -- 172.21.15.61:0/285574618 >> 172.21.15.61:0/285574618 conn(0x7f7c2c0f6250 msgr2=0x7f7c2c0f7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:56.911 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.910+0000 7f7c2bfff700 1 -- 172.21.15.61:0/285574618 shutdown_connections 2023-12-25T21:06:56.912 INFO:teuthology.orchestra.run.smithi061.stderr:/usr/bin/ceph: 2023-12-25T21:06:56.910+0000 7f7c2bfff700 1 -- 172.21.15.61:0/285574618 wait complete. 2023-12-25T21:06:57.205 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:56 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:55.735586+0000 mgr.smithi061.jxxeke (mgr.14164) 4 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T21:06:57.205 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:56 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:55.981150+0000 mon.smithi061 (mon.0) 107 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' 2023-12-25T21:06:57.205 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:56 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:56.908972+0000 mon.smithi061 (mon.0) 108 : audit [DBG] from='client.? 172.21.15.61:0/285574618' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-12-25T21:06:57.300 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-25T21:06:57.309 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: disabled 2023-12-25T21:06:57.310 INFO:teuthology.orchestra.run.smithi061.stderr:Running command: systemctl is-active firewalld.service 2023-12-25T21:06:57.317 INFO:teuthology.orchestra.run.smithi061.stderr:systemctl: inactive 2023-12-25T21:06:57.318 INFO:teuthology.orchestra.run.smithi061.stderr:firewalld.service is not enabled 2023-12-25T21:06:57.318 INFO:teuthology.orchestra.run.smithi061.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-12-25T21:06:57.319 INFO:teuthology.orchestra.run.smithi061.stderr:Ceph Dashboard is now available at: 2023-12-25T21:06:57.320 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-12-25T21:06:57.320 INFO:teuthology.orchestra.run.smithi061.stderr: URL: https://smithi061.front.sepia.ceph.com:8443/ 2023-12-25T21:06:57.320 INFO:teuthology.orchestra.run.smithi061.stderr: User: admin 2023-12-25T21:06:57.320 INFO:teuthology.orchestra.run.smithi061.stderr: Password: lv2kx0o40w 2023-12-25T21:06:57.320 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-12-25T21:06:57.321 INFO:teuthology.orchestra.run.smithi061.stderr:You can access the Ceph CLI with: 2023-12-25T21:06:57.321 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-12-25T21:06:57.321 INFO:teuthology.orchestra.run.smithi061.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 5532d712-a369-11ee-95a5-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-12-25T21:06:57.321 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-12-25T21:06:57.321 INFO:teuthology.orchestra.run.smithi061.stderr:Please consider enabling telemetry to help improve Ceph: 2023-12-25T21:06:57.321 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-12-25T21:06:57.321 INFO:teuthology.orchestra.run.smithi061.stderr: ceph telemetry on 2023-12-25T21:06:57.321 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-12-25T21:06:57.322 INFO:teuthology.orchestra.run.smithi061.stderr:For more information see: 2023-12-25T21:06:57.322 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-12-25T21:06:57.322 INFO:teuthology.orchestra.run.smithi061.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-12-25T21:06:57.322 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-12-25T21:06:57.322 INFO:teuthology.orchestra.run.smithi061.stderr:Bootstrap complete. 2023-12-25T21:06:57.322 INFO:teuthology.orchestra.run.smithi061.stderr:Releasing lock 139968974834152 on /run/cephadm/5532d712-a369-11ee-95a5-87774f69a715.lock 2023-12-25T21:06:57.322 INFO:teuthology.orchestra.run.smithi061.stderr:Lock 139968974834152 released on /run/cephadm/5532d712-a369-11ee-95a5-87774f69a715.lock 2023-12-25T21:06:57.353 INFO:tasks.cephadm:Fetching config... 2023-12-25T21:06:57.354 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-12-25T21:06:57.354 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-12-25T21:06:57.412 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-12-25T21:06:57.412 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-12-25T21:06:57.412 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-12-25T21:06:57.470 INFO:tasks.cephadm:Fetching mon keyring... 2023-12-25T21:06:57.470 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-12-25T21:06:57.470 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/mon.smithi061/keyring of=/dev/stdout 2023-12-25T21:06:57.538 INFO:tasks.cephadm:Fetching pub ssh key... 2023-12-25T21:06:57.538 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-12-25T21:06:57.538 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-12-25T21:06:57.597 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-12-25T21:06:57.597 DEBUG:teuthology.orchestra.run.smithi061:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDF1CLZFIijCNuXLHAYeAVzgtdpgjqt+gv9o3wbHv7twCQ2/+pUUZS1i/ATYzdlLQRQ+vmGLLi31BwaDPxIPJVqOKm2eZkiQmwKkNZ/7UmnBXoFgB/do8+StfyNQXxjLYiupT7nrwOp8ZpEDAQAcz4vAK8SUHvbkLnWn7Lk1nwkU8CG1GRDwYVWPGCWkGaSXc2oto0Xf1nZbsx2JKqJ5copwy3BKzOGY4m+0OuVL2phf0ST709MzY3Q3mtsnW2T0SjwGINQltMI/4ukXFXhIwxQSO3+5DCaC19C8LJCcqpSOz8/Zr0x9P43a6FgzWbWtpwsZNHbxcqktHC9u8UWLDi9Hp6E2r3UzTihJ1FVUWTvwMf85ol/EtLSssQJKzJ0E5rZ7D8/tpFKRKhb76I5VXR59NJhMqwEzUMNaU6ewQUnbz0UjvwK6ysjpI0dwpX+VY4KeaWI854H3Uwfzt+WvHduM3K0Q0yr2tbZscLREGw92M8i9oa7t5mP9Ntc7tgHI+8= ceph-5532d712-a369-11ee-95a5-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-25T21:06:57.673 INFO:teuthology.orchestra.run.smithi061.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDF1CLZFIijCNuXLHAYeAVzgtdpgjqt+gv9o3wbHv7twCQ2/+pUUZS1i/ATYzdlLQRQ+vmGLLi31BwaDPxIPJVqOKm2eZkiQmwKkNZ/7UmnBXoFgB/do8+StfyNQXxjLYiupT7nrwOp8ZpEDAQAcz4vAK8SUHvbkLnWn7Lk1nwkU8CG1GRDwYVWPGCWkGaSXc2oto0Xf1nZbsx2JKqJ5copwy3BKzOGY4m+0OuVL2phf0ST709MzY3Q3mtsnW2T0SjwGINQltMI/4ukXFXhIwxQSO3+5DCaC19C8LJCcqpSOz8/Zr0x9P43a6FgzWbWtpwsZNHbxcqktHC9u8UWLDi9Hp6E2r3UzTihJ1FVUWTvwMf85ol/EtLSssQJKzJ0E5rZ7D8/tpFKRKhb76I5VXR59NJhMqwEzUMNaU6ewQUnbz0UjvwK6ysjpI0dwpX+VY4KeaWI854H3Uwfzt+WvHduM3K0Q0yr2tbZscLREGw92M8i9oa7t5mP9Ntc7tgHI+8= ceph-5532d712-a369-11ee-95a5-87774f69a715 2023-12-25T21:06:57.686 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDF1CLZFIijCNuXLHAYeAVzgtdpgjqt+gv9o3wbHv7twCQ2/+pUUZS1i/ATYzdlLQRQ+vmGLLi31BwaDPxIPJVqOKm2eZkiQmwKkNZ/7UmnBXoFgB/do8+StfyNQXxjLYiupT7nrwOp8ZpEDAQAcz4vAK8SUHvbkLnWn7Lk1nwkU8CG1GRDwYVWPGCWkGaSXc2oto0Xf1nZbsx2JKqJ5copwy3BKzOGY4m+0OuVL2phf0ST709MzY3Q3mtsnW2T0SjwGINQltMI/4ukXFXhIwxQSO3+5DCaC19C8LJCcqpSOz8/Zr0x9P43a6FgzWbWtpwsZNHbxcqktHC9u8UWLDi9Hp6E2r3UzTihJ1FVUWTvwMf85ol/EtLSssQJKzJ0E5rZ7D8/tpFKRKhb76I5VXR59NJhMqwEzUMNaU6ewQUnbz0UjvwK6ysjpI0dwpX+VY4KeaWI854H3Uwfzt+WvHduM3K0Q0yr2tbZscLREGw92M8i9oa7t5mP9Ntc7tgHI+8= ceph-5532d712-a369-11ee-95a5-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-25T21:06:57.733 INFO:teuthology.orchestra.run.smithi174.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDF1CLZFIijCNuXLHAYeAVzgtdpgjqt+gv9o3wbHv7twCQ2/+pUUZS1i/ATYzdlLQRQ+vmGLLi31BwaDPxIPJVqOKm2eZkiQmwKkNZ/7UmnBXoFgB/do8+StfyNQXxjLYiupT7nrwOp8ZpEDAQAcz4vAK8SUHvbkLnWn7Lk1nwkU8CG1GRDwYVWPGCWkGaSXc2oto0Xf1nZbsx2JKqJ5copwy3BKzOGY4m+0OuVL2phf0ST709MzY3Q3mtsnW2T0SjwGINQltMI/4ukXFXhIwxQSO3+5DCaC19C8LJCcqpSOz8/Zr0x9P43a6FgzWbWtpwsZNHbxcqktHC9u8UWLDi9Hp6E2r3UzTihJ1FVUWTvwMf85ol/EtLSssQJKzJ0E5rZ7D8/tpFKRKhb76I5VXR59NJhMqwEzUMNaU6ewQUnbz0UjvwK6ysjpI0dwpX+VY4KeaWI854H3Uwfzt+WvHduM3K0Q0yr2tbZscLREGw92M8i9oa7t5mP9Ntc7tgHI+8= ceph-5532d712-a369-11ee-95a5-87774f69a715 2023-12-25T21:06:57.745 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-12-25T21:06:58.953 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.951+0000 7f8692966700 1 -- 172.21.15.61:0/4028573270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f868c0f51a0 msgr2=0x7f868c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:58.953 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.951+0000 7f8692966700 1 --2- 172.21.15.61:0/4028573270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f868c0f51a0 0x7f868c0f55c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f8688009fb0 tx=0x7f8688009b30).stop 2023-12-25T21:06:58.954 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.951+0000 7f8692966700 1 -- 172.21.15.61:0/4028573270 shutdown_connections 2023-12-25T21:06:58.954 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.951+0000 7f8692966700 1 --2- 172.21.15.61:0/4028573270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f868c0f51a0 0x7f868c0f55c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:58.954 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.951+0000 7f8692966700 1 -- 172.21.15.61:0/4028573270 >> 172.21.15.61:0/4028573270 conn(0x7f868c0f09a0 msgr2=0x7f868c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:58.954 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.951+0000 7f8692966700 1 -- 172.21.15.61:0/4028573270 shutdown_connections 2023-12-25T21:06:58.954 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.951+0000 7f8692966700 1 -- 172.21.15.61:0/4028573270 wait complete. 2023-12-25T21:06:58.954 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.952+0000 7f8692966700 1 Processor -- start 2023-12-25T21:06:58.954 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.952+0000 7f8692966700 1 -- start start 2023-12-25T21:06:58.954 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.952+0000 7f8692966700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f868c0f51a0 0x7f868c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:58.954 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.952+0000 7f8692966700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868c060220 con 0x7f868c0f51a0 2023-12-25T21:06:58.954 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.952+0000 7f8691964700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f868c0f51a0 0x7f868c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:58.954 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.952+0000 7f8691964700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f868c0f51a0 0x7f868c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:40872/0 (socket says 172.21.15.61:40872) 2023-12-25T21:06:58.954 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.952+0000 7f8691964700 1 -- 172.21.15.61:0/854339050 learned_addr learned my addr 172.21.15.61:0/854339050 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:06:58.954 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.953+0000 7f8691964700 1 -- 172.21.15.61:0/854339050 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8688005040 con 0x7f868c0f51a0 2023-12-25T21:06:58.954 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.953+0000 7f8691964700 1 --2- 172.21.15.61:0/854339050 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f868c0f51a0 0x7f868c1091b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f8688002040 tx=0x7f868800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:58.955 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.953+0000 7f867e7fc700 1 -- 172.21.15.61:0/854339050 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86880095f0 con 0x7f868c0f51a0 2023-12-25T21:06:58.955 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.953+0000 7f867e7fc700 1 -- 172.21.15.61:0/854339050 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8688007e30 con 0x7f868c0f51a0 2023-12-25T21:06:58.955 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.953+0000 7f867e7fc700 1 -- 172.21.15.61:0/854339050 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8688014d20 con 0x7f868c0f51a0 2023-12-25T21:06:58.955 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.953+0000 7f8692966700 1 -- 172.21.15.61:0/854339050 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f868c060420 con 0x7f868c0f51a0 2023-12-25T21:06:58.955 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.953+0000 7f8692966700 1 -- 172.21.15.61:0/854339050 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f868c109860 con 0x7f868c0f51a0 2023-12-25T21:06:58.956 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.954+0000 7f867e7fc700 1 -- 172.21.15.61:0/854339050 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8688025070 con 0x7f868c0f51a0 2023-12-25T21:06:58.956 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.954+0000 7f867e7fc700 1 --2- 172.21.15.61:0/854339050 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f867802ffb0 0x7f8678032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:06:58.956 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.954+0000 7f867e7fc700 1 -- 172.21.15.61:0/854339050 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f86880408d0 con 0x7f868c0f51a0 2023-12-25T21:06:58.956 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.954+0000 7f8691163700 1 --2- 172.21.15.61:0/854339050 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f867802ffb0 0x7f8678032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:06:58.956 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.954+0000 7f8692966700 1 -- 172.21.15.61:0/854339050 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f868c059100 con 0x7f868c0f51a0 2023-12-25T21:06:58.956 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.954+0000 7f8691163700 1 --2- 172.21.15.61:0/854339050 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f867802ffb0 0x7f8678032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f86800096a0 tx=0x7f8680006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:06:58.958 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:58.958+0000 7f867e7fc700 1 -- 172.21.15.61:0/854339050 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f868800f4e0 con 0x7f868c0f51a0 2023-12-25T21:06:59.117 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:59.117+0000 7f8692966700 1 -- 172.21.15.61:0/854339050 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f868c045ad0 con 0x7f868c0f51a0 2023-12-25T21:06:59.120 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:59.119+0000 7f867e7fc700 1 -- 172.21.15.61:0/854339050 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v10)=0 v10) v1 ==== 125+0+0 (secure 0 0 0) 0x7f868800f300 con 0x7f868c0f51a0 2023-12-25T21:06:59.122 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:59.121+0000 7f8692966700 1 -- 172.21.15.61:0/854339050 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f867802ffb0 msgr2=0x7f8678032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:59.122 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:59.121+0000 7f8692966700 1 --2- 172.21.15.61:0/854339050 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f867802ffb0 0x7f8678032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f86800096a0 tx=0x7f8680006b40).stop 2023-12-25T21:06:59.122 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:59.121+0000 7f8692966700 1 -- 172.21.15.61:0/854339050 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f868c0f51a0 msgr2=0x7f868c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:06:59.122 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:59.121+0000 7f8692966700 1 --2- 172.21.15.61:0/854339050 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f868c0f51a0 0x7f868c1091b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f8688002040 tx=0x7f868800b040).stop 2023-12-25T21:06:59.122 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:59.121+0000 7f8692966700 1 -- 172.21.15.61:0/854339050 shutdown_connections 2023-12-25T21:06:59.122 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:59.121+0000 7f8692966700 1 --2- 172.21.15.61:0/854339050 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f867802ffb0 0x7f8678032470 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:59.122 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:59.121+0000 7f8692966700 1 --2- 172.21.15.61:0/854339050 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f868c0f51a0 0x7f868c1091b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:06:59.122 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:59.121+0000 7f8692966700 1 -- 172.21.15.61:0/854339050 >> 172.21.15.61:0/854339050 conn(0x7f868c0f09a0 msgr2=0x7f868c0f14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:06:59.122 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:59.121+0000 7f8692966700 1 -- 172.21.15.61:0/854339050 shutdown_connections 2023-12-25T21:06:59.122 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:06:59.122+0000 7f8692966700 1 -- 172.21.15.61:0/854339050 wait complete. 2023-12-25T21:06:59.523 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi174 2023-12-25T21:06:59.524 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-12-25T21:06:59.524 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.conf 2023-12-25T21:06:59.545 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-12-25T21:06:59.546 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-25T21:06:59.607 INFO:tasks.cephadm:Adding host smithi174 to orchestrator... 2023-12-25T21:06:59.607 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph orch host add smithi174 2023-12-25T21:06:59.957 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:59 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:06:58.599153+0000 mon.smithi061 (mon.0) 109 : cluster [DBG] mgrmap e17: smithi061.jxxeke(active, since 6s) 2023-12-25T21:06:59.957 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:06:59 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:06:59.120683+0000 mon.smithi061 (mon.0) 110 : audit [INF] from='client.? 172.21.15.61:0/854339050' entity='client.admin' 2023-12-25T21:07:02.104 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:00.763992+0000 mon.smithi061 (mon.0) 111 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' 2023-12-25T21:07:02.104 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:00.988519+0000 mon.smithi061 (mon.0) 112 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' 2023-12-25T21:07:02.104 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:01.642287+0000 mon.smithi061 (mon.0) 113 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' 2023-12-25T21:07:05.455 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:05 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:04.028864+0000 mon.smithi061 (mon.0) 114 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' 2023-12-25T21:07:05.455 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:05 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:04.317277+0000 mon.smithi061 (mon.0) 115 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' 2023-12-25T21:07:05.456 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:05 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:04.319353+0000 mon.smithi061 (mon.0) 116 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' 2023-12-25T21:07:06.455 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:06 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cephadm 2023-12-25T21:07:04.325909+0000 mgr.smithi061.jxxeke (mgr.14164) 5 : cephadm [INF] Deploying daemon alertmanager.smithi061 on smithi061 2023-12-25T21:07:13.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:13 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:12.567366+0000 mgr.smithi061.jxxeke (mgr.14164) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:15.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:15 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:14.567708+0000 mgr.smithi061.jxxeke (mgr.14164) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:18.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:18 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:16.567986+0000 mgr.smithi061.jxxeke (mgr.14164) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:20.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:20 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:18.568302+0000 mgr.smithi061.jxxeke (mgr.14164) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:22.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.031+0000 7f61d6890700 1 -- 172.21.15.174:0/3499506805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61d00f62a0 msgr2=0x7f61d00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:22.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.031+0000 7f61d6890700 1 --2- 172.21.15.174:0/3499506805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61d00f62a0 0x7f61d00f66c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f61c0009fb0 tx=0x7f61c0009b30).stop 2023-12-25T21:07:22.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.031+0000 7f61d6890700 1 -- 172.21.15.174:0/3499506805 shutdown_connections 2023-12-25T21:07:22.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.031+0000 7f61d6890700 1 --2- 172.21.15.174:0/3499506805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61d00f62a0 0x7f61d00f66c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:22.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.031+0000 7f61d6890700 1 -- 172.21.15.174:0/3499506805 >> 172.21.15.174:0/3499506805 conn(0x7f61d00f1aa0 msgr2=0x7f61d00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:22.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.032+0000 7f61d6890700 1 -- 172.21.15.174:0/3499506805 shutdown_connections 2023-12-25T21:07:22.035 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.032+0000 7f61d6890700 1 -- 172.21.15.174:0/3499506805 wait complete. 2023-12-25T21:07:22.035 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.032+0000 7f61d6890700 1 Processor -- start 2023-12-25T21:07:22.035 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.032+0000 7f61d6890700 1 -- start start 2023-12-25T21:07:22.035 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.032+0000 7f61d6890700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61d00f62a0 0x7f61d01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:22.035 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.032+0000 7f61d6890700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d01069a0 con 0x7f61d00f62a0 2023-12-25T21:07:22.035 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.032+0000 7f61d588e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61d00f62a0 0x7f61d01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:22.035 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.032+0000 7f61d588e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61d00f62a0 0x7f61d01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:43656/0 (socket says 172.21.15.174:43656) 2023-12-25T21:07:22.035 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.032+0000 7f61d588e700 1 -- 172.21.15.174:0/1393545853 learned_addr learned my addr 172.21.15.174:0/1393545853 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:07:22.035 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.033+0000 7f61d588e700 1 -- 172.21.15.174:0/1393545853 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c0005040 con 0x7f61d00f62a0 2023-12-25T21:07:22.035 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.033+0000 7f61d588e700 1 --2- 172.21.15.174:0/1393545853 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61d00f62a0 0x7f61d01064b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f61c0013510 tx=0x7f61c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:22.035 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.033+0000 7f61c67fc700 1 -- 172.21.15.174:0/1393545853 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61c0009320 con 0x7f61d00f62a0 2023-12-25T21:07:22.035 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.033+0000 7f61c67fc700 1 -- 172.21.15.174:0/1393545853 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61c0013f00 con 0x7f61d00f62a0 2023-12-25T21:07:22.036 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.033+0000 7f61d6890700 1 -- 172.21.15.174:0/1393545853 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d0106ba0 con 0x7f61d00f62a0 2023-12-25T21:07:22.036 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.033+0000 7f61c67fc700 1 -- 172.21.15.174:0/1393545853 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61c0014dc0 con 0x7f61d00f62a0 2023-12-25T21:07:22.036 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.033+0000 7f61d6890700 1 -- 172.21.15.174:0/1393545853 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d0107030 con 0x7f61d00f62a0 2023-12-25T21:07:22.036 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.034+0000 7f61c67fc700 1 -- 172.21.15.174:0/1393545853 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61c00095f0 con 0x7f61d00f62a0 2023-12-25T21:07:22.037 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.034+0000 7f61d6890700 1 -- 172.21.15.174:0/1393545853 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d0045ad0 con 0x7f61d00f62a0 2023-12-25T21:07:22.037 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.034+0000 7f61c67fc700 1 --2- 172.21.15.174:0/1393545853 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f61bc02ff60 0x7f61bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:22.037 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.034+0000 7f61c67fc700 1 -- 172.21.15.174:0/1393545853 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61c0041100 con 0x7f61d00f62a0 2023-12-25T21:07:22.037 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.034+0000 7f61d508d700 1 --2- 172.21.15.174:0/1393545853 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f61bc02ff60 0x7f61bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:22.037 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.035+0000 7f61d508d700 1 --2- 172.21.15.174:0/1393545853 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f61bc02ff60 0x7f61bc032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f61cc0096a0 tx=0x7f61cc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:22.040 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.039+0000 7f61c67fc700 1 -- 172.21.15.174:0/1393545853 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61c0014420 con 0x7f61d00f62a0 2023-12-25T21:07:22.209 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.208+0000 7f61d6890700 1 -- 172.21.15.174:0/1393545853 --> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}) v1 -- 0x7f61d00f2c20 con 0x7f61bc02ff60 2023-12-25T21:07:22.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:22 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:20.568622+0000 mgr.smithi061.jxxeke (mgr.14164) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:22.816 INFO:teuthology.orchestra.run.smithi174.stdout:Added host 'smithi174' 2023-12-25T21:07:22.816 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.814+0000 7f61c67fc700 1 -- 172.21.15.174:0/1393545853 <== mgr.14164 v2:172.21.15.61:6800/889227411 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f61d00f2c20 con 0x7f61bc02ff60 2023-12-25T21:07:22.817 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.816+0000 7f61d6890700 1 -- 172.21.15.174:0/1393545853 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f61bc02ff60 msgr2=0x7f61bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:22.817 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.816+0000 7f61d6890700 1 --2- 172.21.15.174:0/1393545853 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f61bc02ff60 0x7f61bc032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f61cc0096a0 tx=0x7f61cc006b40).stop 2023-12-25T21:07:22.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.817+0000 7f61d6890700 1 -- 172.21.15.174:0/1393545853 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61d00f62a0 msgr2=0x7f61d01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:22.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.817+0000 7f61d6890700 1 --2- 172.21.15.174:0/1393545853 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61d00f62a0 0x7f61d01064b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f61c0013510 tx=0x7f61c000b040).stop 2023-12-25T21:07:22.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.817+0000 7f61d6890700 1 -- 172.21.15.174:0/1393545853 shutdown_connections 2023-12-25T21:07:22.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.817+0000 7f61d6890700 1 --2- 172.21.15.174:0/1393545853 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f61bc02ff60 0x7f61bc032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:22.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.817+0000 7f61d6890700 1 --2- 172.21.15.174:0/1393545853 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61d00f62a0 0x7f61d01064b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:22.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.817+0000 7f61d6890700 1 -- 172.21.15.174:0/1393545853 >> 172.21.15.174:0/1393545853 conn(0x7f61d00f1aa0 msgr2=0x7f61d00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:22.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.817+0000 7f61d6890700 1 -- 172.21.15.174:0/1393545853 shutdown_connections 2023-12-25T21:07:22.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:22.817+0000 7f61d6890700 1 -- 172.21.15.174:0/1393545853 wait complete. 2023-12-25T21:07:23.187 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph orch host ls --format=json 2023-12-25T21:07:23.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:22.210505+0000 mgr.smithi061.jxxeke (mgr.14164) 11 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T21:07:23.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:22.814852+0000 mon.smithi061 (mon.0) 117 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' 2023-12-25T21:07:24.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.269+0000 7fe8fea1f700 1 -- 172.21.15.174:0/3613687876 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f51a0 msgr2=0x7fe8f80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:24.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.269+0000 7fe8fea1f700 1 --2- 172.21.15.174:0/3613687876 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f51a0 0x7fe8f80f55c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fe8e8009fb0 tx=0x7fe8e8009b30).stop 2023-12-25T21:07:24.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.269+0000 7fe8fea1f700 1 -- 172.21.15.174:0/3613687876 shutdown_connections 2023-12-25T21:07:24.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.269+0000 7fe8fea1f700 1 --2- 172.21.15.174:0/3613687876 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f51a0 0x7fe8f80f55c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:24.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.269+0000 7fe8fea1f700 1 -- 172.21.15.174:0/3613687876 >> 172.21.15.174:0/3613687876 conn(0x7fe8f80f09a0 msgr2=0x7fe8f80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:24.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.270+0000 7fe8fea1f700 1 -- 172.21.15.174:0/3613687876 shutdown_connections 2023-12-25T21:07:24.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.270+0000 7fe8fea1f700 1 -- 172.21.15.174:0/3613687876 wait complete. 2023-12-25T21:07:24.273 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.270+0000 7fe8fea1f700 1 Processor -- start 2023-12-25T21:07:24.273 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.270+0000 7fe8fea1f700 1 -- start start 2023-12-25T21:07:24.273 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.270+0000 7fe8fea1f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f51a0 0x7fe8f81064f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:24.273 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.270+0000 7fe8fea1f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f81069e0 con 0x7fe8f80f51a0 2023-12-25T21:07:24.273 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.270+0000 7fe8fda1d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f51a0 0x7fe8f81064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:24.273 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.270+0000 7fe8fda1d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f51a0 0x7fe8f81064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:43676/0 (socket says 172.21.15.174:43676) 2023-12-25T21:07:24.273 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.270+0000 7fe8fda1d700 1 -- 172.21.15.174:0/3367314550 learned_addr learned my addr 172.21.15.174:0/3367314550 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:07:24.273 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.271+0000 7fe8fda1d700 1 -- 172.21.15.174:0/3367314550 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e8005040 con 0x7fe8f80f51a0 2023-12-25T21:07:24.273 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.271+0000 7fe8fda1d700 1 --2- 172.21.15.174:0/3367314550 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f51a0 0x7fe8f81064f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fe8e8009830 tx=0x7fe8e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:24.273 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.271+0000 7fe8ee7fc700 1 -- 172.21.15.174:0/3367314550 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8e8009320 con 0x7fe8f80f51a0 2023-12-25T21:07:24.273 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.271+0000 7fe8fea1f700 1 -- 172.21.15.174:0/3367314550 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f8106be0 con 0x7fe8f80f51a0 2023-12-25T21:07:24.273 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.271+0000 7fe8ee7fc700 1 -- 172.21.15.174:0/3367314550 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8e80136c0 con 0x7fe8f80f51a0 2023-12-25T21:07:24.274 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.271+0000 7fe8ee7fc700 1 -- 172.21.15.174:0/3367314550 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8e801ee90 con 0x7fe8f80f51a0 2023-12-25T21:07:24.274 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.271+0000 7fe8fea1f700 1 -- 172.21.15.174:0/3367314550 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f8107070 con 0x7fe8f80f51a0 2023-12-25T21:07:24.274 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.272+0000 7fe8ee7fc700 1 -- 172.21.15.174:0/3367314550 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8e801e420 con 0x7fe8f80f51a0 2023-12-25T21:07:24.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.272+0000 7fe8fea1f700 1 -- 172.21.15.174:0/3367314550 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f8045ad0 con 0x7fe8f80f51a0 2023-12-25T21:07:24.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.272+0000 7fe8ee7fc700 1 --2- 172.21.15.174:0/3367314550 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe8e402ff10 0x7fe8e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:24.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.272+0000 7fe8ee7fc700 1 -- 172.21.15.174:0/3367314550 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe8e8041040 con 0x7fe8f80f51a0 2023-12-25T21:07:24.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.272+0000 7fe8fd21c700 1 --2- 172.21.15.174:0/3367314550 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe8e402ff10 0x7fe8e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:24.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.273+0000 7fe8fd21c700 1 --2- 172.21.15.174:0/3367314550 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe8e402ff10 0x7fe8e40323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe8f40096a0 tx=0x7fe8f4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:24.277 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.277+0000 7fe8ee7fc700 1 -- 172.21.15.174:0/3367314550 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe8e806d2b0 con 0x7fe8f80f51a0 2023-12-25T21:07:24.448 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.447+0000 7fe8fea1f700 1 -- 172.21.15.174:0/3367314550 --> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe8f81078d0 con 0x7fe8e402ff10 2023-12-25T21:07:24.450 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.450+0000 7fe8ee7fc700 1 -- 172.21.15.174:0/3367314550 <== mgr.14164 v2:172.21.15.61:6800/889227411 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7fe8f81078d0 con 0x7fe8e402ff10 2023-12-25T21:07:24.450 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:07:24.450 INFO:teuthology.orchestra.run.smithi174.stdout:[{"addr": "172.21.15.61", "hostname": "smithi061", "labels": [], "status": ""}, {"addr": "smithi174", "hostname": "smithi174", "labels": [], "status": ""}] 2023-12-25T21:07:24.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.451+0000 7fe8fea1f700 1 -- 172.21.15.174:0/3367314550 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe8e402ff10 msgr2=0x7fe8e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:24.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.451+0000 7fe8fea1f700 1 --2- 172.21.15.174:0/3367314550 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe8e402ff10 0x7fe8e40323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe8f40096a0 tx=0x7fe8f4006b40).stop 2023-12-25T21:07:24.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.451+0000 7fe8fea1f700 1 -- 172.21.15.174:0/3367314550 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f51a0 msgr2=0x7fe8f81064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:24.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.451+0000 7fe8fea1f700 1 --2- 172.21.15.174:0/3367314550 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f51a0 0x7fe8f81064f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fe8e8009830 tx=0x7fe8e800b040).stop 2023-12-25T21:07:24.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.451+0000 7fe8fea1f700 1 -- 172.21.15.174:0/3367314550 shutdown_connections 2023-12-25T21:07:24.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.451+0000 7fe8fea1f700 1 --2- 172.21.15.174:0/3367314550 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe8e402ff10 0x7fe8e40323d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:24.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.451+0000 7fe8fea1f700 1 --2- 172.21.15.174:0/3367314550 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8f80f51a0 0x7fe8f81064f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:24.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.451+0000 7fe8fea1f700 1 -- 172.21.15.174:0/3367314550 >> 172.21.15.174:0/3367314550 conn(0x7fe8f80f09a0 msgr2=0x7fe8f805fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:24.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.451+0000 7fe8fea1f700 1 -- 172.21.15.174:0/3367314550 shutdown_connections 2023-12-25T21:07:24.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:24.451+0000 7fe8fea1f700 1 -- 172.21.15.174:0/3367314550 wait complete. 2023-12-25T21:07:24.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:24 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:22.568909+0000 mgr.smithi061.jxxeke (mgr.14164) 12 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:24.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:24 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cephadm 2023-12-25T21:07:22.815276+0000 mgr.smithi061.jxxeke (mgr.14164) 13 : cephadm [INF] Added host smithi174 2023-12-25T21:07:24.825 INFO:tasks.cephadm:Setting crush tunables to default 2023-12-25T21:07:24.826 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph osd crush tunables default 2023-12-25T21:07:26.078 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.074+0000 7f4826f2b700 1 -- 172.21.15.61:0/2102364865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48200f78e0 msgr2=0x7f48200f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:26.078 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.074+0000 7f4826f2b700 1 --2- 172.21.15.61:0/2102364865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48200f78e0 0x7f48200f7d00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f481c009fb0 tx=0x7f481c009b30).stop 2023-12-25T21:07:26.078 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.074+0000 7f4826f2b700 1 -- 172.21.15.61:0/2102364865 shutdown_connections 2023-12-25T21:07:26.078 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.074+0000 7f4826f2b700 1 --2- 172.21.15.61:0/2102364865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48200f78e0 0x7f48200f7d00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:26.078 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.075+0000 7f4826f2b700 1 -- 172.21.15.61:0/2102364865 >> 172.21.15.61:0/2102364865 conn(0x7f48200f0120 msgr2=0x7f48200f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:26.078 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.075+0000 7f4826f2b700 1 -- 172.21.15.61:0/2102364865 shutdown_connections 2023-12-25T21:07:26.078 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.075+0000 7f4826f2b700 1 -- 172.21.15.61:0/2102364865 wait complete. 2023-12-25T21:07:26.078 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.075+0000 7f4826f2b700 1 Processor -- start 2023-12-25T21:07:26.078 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.075+0000 7f4826f2b700 1 -- start start 2023-12-25T21:07:26.078 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.075+0000 7f4826f2b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48200f78e0 0x7f48201064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:26.079 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.075+0000 7f4826f2b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48201069d0 con 0x7f48200f78e0 2023-12-25T21:07:26.079 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.075+0000 7f4825f29700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48200f78e0 0x7f48201064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:26.079 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.075+0000 7f4825f29700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48200f78e0 0x7f48201064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:35436/0 (socket says 172.21.15.61:35436) 2023-12-25T21:07:26.079 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.075+0000 7f4825f29700 1 -- 172.21.15.61:0/4081518772 learned_addr learned my addr 172.21.15.61:0/4081518772 (peer_addr_for_me v2:172.21.15.61:0/0) 2023-12-25T21:07:26.079 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.076+0000 7f4825f29700 1 -- 172.21.15.61:0/4081518772 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c005040 con 0x7f48200f78e0 2023-12-25T21:07:26.079 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.076+0000 7f4825f29700 1 --2- 172.21.15.61:0/4081518772 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48200f78e0 0x7f48201064e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f481c009830 tx=0x7f481c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:26.079 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.076+0000 7f48127fc700 1 -- 172.21.15.61:0/4081518772 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f481c009320 con 0x7f48200f78e0 2023-12-25T21:07:26.079 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.076+0000 7f48127fc700 1 -- 172.21.15.61:0/4081518772 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f481c0136c0 con 0x7f48200f78e0 2023-12-25T21:07:26.079 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.076+0000 7f4826f2b700 1 -- 172.21.15.61:0/4081518772 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4820106bd0 con 0x7f48200f78e0 2023-12-25T21:07:26.079 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.076+0000 7f48127fc700 1 -- 172.21.15.61:0/4081518772 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f481c01ee90 con 0x7f48200f78e0 2023-12-25T21:07:26.079 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.076+0000 7f4826f2b700 1 -- 172.21.15.61:0/4081518772 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4820107060 con 0x7f48200f78e0 2023-12-25T21:07:26.080 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.077+0000 7f48127fc700 1 -- 172.21.15.61:0/4081518772 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f481c01e420 con 0x7f48200f78e0 2023-12-25T21:07:26.080 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.077+0000 7f48127fc700 1 --2- 172.21.15.61:0/4081518772 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f480c02ff10 0x7f480c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:26.080 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.077+0000 7f4826f2b700 1 -- 172.21.15.61:0/4081518772 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4820045ad0 con 0x7f48200f78e0 2023-12-25T21:07:26.080 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.077+0000 7f48127fc700 1 -- 172.21.15.61:0/4081518772 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f481c040f10 con 0x7f48200f78e0 2023-12-25T21:07:26.080 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.077+0000 7f4825728700 1 --2- 172.21.15.61:0/4081518772 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f480c02ff10 0x7f480c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:26.081 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.078+0000 7f4825728700 1 --2- 172.21.15.61:0/4081518772 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f480c02ff10 0x7f480c0323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f48140096a0 tx=0x7f4814006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:26.083 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.081+0000 7f48127fc700 1 -- 172.21.15.61:0/4081518772 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f481c0095f0 con 0x7f48200f78e0 2023-12-25T21:07:26.246 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.244+0000 7f4826f2b700 1 -- 172.21.15.61:0/4081518772 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f4820107f10 con 0x7f48200f78e0 2023-12-25T21:07:26.289 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.288+0000 7f48127fc700 1 -- 172.21.15.61:0/4081518772 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f481c00e2a0 con 0x7f48200f78e0 2023-12-25T21:07:26.292 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.290+0000 7f4826f2b700 1 -- 172.21.15.61:0/4081518772 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f480c02ff10 msgr2=0x7f480c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:26.292 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.290+0000 7f4826f2b700 1 --2- 172.21.15.61:0/4081518772 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f480c02ff10 0x7f480c0323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f48140096a0 tx=0x7f4814006b40).stop 2023-12-25T21:07:26.292 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.290+0000 7f4826f2b700 1 -- 172.21.15.61:0/4081518772 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48200f78e0 msgr2=0x7f48201064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:26.292 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.290+0000 7f4826f2b700 1 --2- 172.21.15.61:0/4081518772 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48200f78e0 0x7f48201064e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f481c009830 tx=0x7f481c00b040).stop 2023-12-25T21:07:26.292 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.290+0000 7f4826f2b700 1 -- 172.21.15.61:0/4081518772 shutdown_connections 2023-12-25T21:07:26.292 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.290+0000 7f4826f2b700 1 --2- 172.21.15.61:0/4081518772 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f480c02ff10 0x7f480c0323d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:26.292 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.290+0000 7f4826f2b700 1 --2- 172.21.15.61:0/4081518772 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48200f78e0 0x7f48201064e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:26.292 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.290+0000 7f4826f2b700 1 -- 172.21.15.61:0/4081518772 >> 172.21.15.61:0/4081518772 conn(0x7f48200f0120 msgr2=0x7f48200f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:26.292 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.290+0000 7f4826f2b700 1 -- 172.21.15.61:0/4081518772 shutdown_connections 2023-12-25T21:07:26.292 INFO:teuthology.orchestra.run.smithi061.stderr:2023-12-25T21:07:26.290+0000 7f4826f2b700 1 -- 172.21.15.61:0/4081518772 wait complete. 2023-12-25T21:07:26.293 INFO:teuthology.orchestra.run.smithi061.stderr:adjusted tunables profile to default 2023-12-25T21:07:26.574 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:26 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:24.450026+0000 mgr.smithi061.jxxeke (mgr.14164) 14 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-25T21:07:26.574 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:26 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:24.569188+0000 mgr.smithi061.jxxeke (mgr.14164) 15 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:26.574 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:26 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:26.246791+0000 mon.smithi061 (mon.0) 118 : audit [INF] from='client.? 172.21.15.61:0/4081518772' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-12-25T21:07:26.675 INFO:tasks.cephadm:Adding mon.smithi061 on smithi061 2023-12-25T21:07:26.675 INFO:tasks.cephadm:Adding mon.smithi174 on smithi174 2023-12-25T21:07:26.675 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph orch apply mon '2;smithi061:172.21.15.61=smithi061;smithi174:172.21.15.174=smithi174' 2023-12-25T21:07:27.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:27 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:26.290200+0000 mon.smithi061 (mon.0) 119 : audit [INF] from='client.? 172.21.15.61:0/4081518772' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-12-25T21:07:27.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:27 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:26.290240+0000 mon.smithi061 (mon.0) 120 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-12-25T21:07:27.787 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.785+0000 7fe73c3ed700 1 -- 172.21.15.174:0/2981204628 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 msgr2=0x7fe7340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:27.787 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.785+0000 7fe73c3ed700 1 --2- 172.21.15.174:0/2981204628 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 0x7fe7340f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fe728009fb0 tx=0x7fe728009b30).stop 2023-12-25T21:07:27.787 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.785+0000 7fe73c3ed700 1 -- 172.21.15.174:0/2981204628 shutdown_connections 2023-12-25T21:07:27.787 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.785+0000 7fe73c3ed700 1 --2- 172.21.15.174:0/2981204628 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 0x7fe7340f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:27.787 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.785+0000 7fe73c3ed700 1 -- 172.21.15.174:0/2981204628 >> 172.21.15.174:0/2981204628 conn(0x7fe7340f0b70 msgr2=0x7fe7340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:27.787 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.785+0000 7fe73c3ed700 1 -- 172.21.15.174:0/2981204628 shutdown_connections 2023-12-25T21:07:27.788 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.785+0000 7fe73c3ed700 1 -- 172.21.15.174:0/2981204628 wait complete. 2023-12-25T21:07:27.788 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.786+0000 7fe73c3ed700 1 Processor -- start 2023-12-25T21:07:27.788 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.786+0000 7fe73c3ed700 1 -- start start 2023-12-25T21:07:27.788 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.786+0000 7fe73c3ed700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 0x7fe734109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:27.788 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.786+0000 7fe73c3ed700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe734060220 con 0x7fe7340f5410 2023-12-25T21:07:27.788 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.786+0000 7fe73a189700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 0x7fe734109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:27.788 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.786+0000 7fe73a189700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 0x7fe734109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:39446/0 (socket says 172.21.15.174:39446) 2023-12-25T21:07:27.788 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.786+0000 7fe73a189700 1 -- 172.21.15.174:0/3486075067 learned_addr learned my addr 172.21.15.174:0/3486075067 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:07:27.789 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.787+0000 7fe73a189700 1 -- 172.21.15.174:0/3486075067 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe728005040 con 0x7fe7340f5410 2023-12-25T21:07:27.790 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.787+0000 7fe73a189700 1 --2- 172.21.15.174:0/3486075067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 0x7fe734109450 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fe728013510 tx=0x7fe72800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:27.790 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.787+0000 7fe726ffd700 1 -- 172.21.15.174:0/3486075067 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe728009320 con 0x7fe7340f5410 2023-12-25T21:07:27.790 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.787+0000 7fe73c3ed700 1 -- 172.21.15.174:0/3486075067 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe734060420 con 0x7fe7340f5410 2023-12-25T21:07:27.790 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.787+0000 7fe726ffd700 1 -- 172.21.15.174:0/3486075067 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe728013f00 con 0x7fe7340f5410 2023-12-25T21:07:27.790 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.787+0000 7fe726ffd700 1 -- 172.21.15.174:0/3486075067 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe728014d90 con 0x7fe7340f5410 2023-12-25T21:07:27.790 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.787+0000 7fe73c3ed700 1 -- 172.21.15.174:0/3486075067 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe734109b00 con 0x7fe7340f5410 2023-12-25T21:07:27.790 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.788+0000 7fe726ffd700 1 -- 172.21.15.174:0/3486075067 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7280095f0 con 0x7fe7340f5410 2023-12-25T21:07:27.790 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.788+0000 7fe73c3ed700 1 -- 172.21.15.174:0/3486075067 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe734059100 con 0x7fe7340f5410 2023-12-25T21:07:27.790 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.788+0000 7fe726ffd700 1 --2- 172.21.15.174:0/3486075067 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe72002ff60 0x7fe720032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:27.790 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.788+0000 7fe726ffd700 1 -- 172.21.15.174:0/3486075067 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe7280410c0 con 0x7fe7340f5410 2023-12-25T21:07:27.790 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.788+0000 7fe739988700 1 --2- 172.21.15.174:0/3486075067 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe72002ff60 0x7fe720032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:27.790 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.789+0000 7fe739988700 1 --2- 172.21.15.174:0/3486075067 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe72002ff60 0x7fe720032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fe7300096a0 tx=0x7fe730006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:27.793 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.793+0000 7fe726ffd700 1 -- 172.21.15.174:0/3486075067 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe728014420 con 0x7fe7340f5410 2023-12-25T21:07:27.969 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.968+0000 7fe73c3ed700 1 -- 172.21.15.174:0/3486075067 --> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi061:172.21.15.61=smithi061;smithi174:172.21.15.174=smithi174", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7340f1e20 con 0x7fe72002ff60 2023-12-25T21:07:27.975 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.975+0000 7fe726ffd700 1 -- 172.21.15.174:0/3486075067 <== mgr.14164 v2:172.21.15.61:6800/889227411 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fe7340f1e20 con 0x7fe72002ff60 2023-12-25T21:07:27.975 INFO:teuthology.orchestra.run.smithi174.stdout:Scheduled mon update... 2023-12-25T21:07:27.976 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.976+0000 7fe73c3ed700 1 -- 172.21.15.174:0/3486075067 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe72002ff60 msgr2=0x7fe720032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:27.977 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.976+0000 7fe73c3ed700 1 --2- 172.21.15.174:0/3486075067 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe72002ff60 0x7fe720032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fe7300096a0 tx=0x7fe730006b40).stop 2023-12-25T21:07:27.977 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.976+0000 7fe73c3ed700 1 -- 172.21.15.174:0/3486075067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 msgr2=0x7fe734109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:27.977 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.976+0000 7fe73c3ed700 1 --2- 172.21.15.174:0/3486075067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 0x7fe734109450 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fe728013510 tx=0x7fe72800b040).stop 2023-12-25T21:07:27.977 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.976+0000 7fe73c3ed700 1 -- 172.21.15.174:0/3486075067 shutdown_connections 2023-12-25T21:07:27.977 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.976+0000 7fe73c3ed700 1 --2- 172.21.15.174:0/3486075067 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe72002ff60 0x7fe720032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:27.978 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.976+0000 7fe73c3ed700 1 --2- 172.21.15.174:0/3486075067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 0x7fe734109450 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:27.978 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.976+0000 7fe73c3ed700 1 -- 172.21.15.174:0/3486075067 >> 172.21.15.174:0/3486075067 conn(0x7fe7340f0b70 msgr2=0x7fe7340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:27.978 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.976+0000 7fe73c3ed700 1 -- 172.21.15.174:0/3486075067 shutdown_connections 2023-12-25T21:07:27.978 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:27.976+0000 7fe73c3ed700 1 -- 172.21.15.174:0/3486075067 wait complete. 2023-12-25T21:07:28.360 DEBUG:teuthology.orchestra.run.smithi174:mon.smithi174> sudo journalctl -f -n 0 -u ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi174.service 2023-12-25T21:07:28.362 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:07:28.362 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:07:28.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:-- Logs begin at Mon 2023-12-25 20:53:27 UTC. -- 2023-12-25T21:07:28.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:28 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:26.569550+0000 mgr.smithi061.jxxeke (mgr.14164) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:28.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:28 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:27.974920+0000 mon.smithi061 (mon.0) 121 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' 2023-12-25T21:07:29.535 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.534+0000 7f752759e700 1 -- 172.21.15.174:0/3434250657 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75280601e0 msgr2=0x7f75280f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:29.535 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.534+0000 7f752759e700 1 --2- 172.21.15.174:0/3434250657 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75280601e0 0x7f75280f8f20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f7518009fb0 tx=0x7f7518009b30).stop 2023-12-25T21:07:29.536 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.534+0000 7f752759e700 1 -- 172.21.15.174:0/3434250657 shutdown_connections 2023-12-25T21:07:29.536 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.534+0000 7f752759e700 1 --2- 172.21.15.174:0/3434250657 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75280601e0 0x7f75280f8f20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:29.536 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.534+0000 7f752759e700 1 -- 172.21.15.174:0/3434250657 >> 172.21.15.174:0/3434250657 conn(0x7f75280f09a0 msgr2=0x7f75280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:29.536 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.534+0000 7f752759e700 1 -- 172.21.15.174:0/3434250657 shutdown_connections 2023-12-25T21:07:29.536 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.534+0000 7f752759e700 1 -- 172.21.15.174:0/3434250657 wait complete. 2023-12-25T21:07:29.536 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.534+0000 7f752759e700 1 Processor -- start 2023-12-25T21:07:29.536 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.534+0000 7f752759e700 1 -- start start 2023-12-25T21:07:29.536 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.535+0000 7f752759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75280601e0 0x7f7528106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:29.536 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.535+0000 7f752759e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75281069f0 con 0x7f75280601e0 2023-12-25T21:07:29.536 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.535+0000 7f752659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75280601e0 0x7f7528106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:29.536 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.535+0000 7f752659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75280601e0 0x7f7528106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:39458/0 (socket says 172.21.15.174:39458) 2023-12-25T21:07:29.536 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.535+0000 7f752659c700 1 -- 172.21.15.174:0/4062507105 learned_addr learned my addr 172.21.15.174:0/4062507105 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:07:29.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.535+0000 7f752659c700 1 -- 172.21.15.174:0/4062507105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7518005040 con 0x7f75280601e0 2023-12-25T21:07:29.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.535+0000 7f752659c700 1 --2- 172.21.15.174:0/4062507105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75280601e0 0x7f7528106500 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f7518013510 tx=0x7f751800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:29.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.535+0000 7f7516ffd700 1 -- 172.21.15.174:0/4062507105 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7518009320 con 0x7f75280601e0 2023-12-25T21:07:29.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.535+0000 7f7516ffd700 1 -- 172.21.15.174:0/4062507105 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7518013f00 con 0x7f75280601e0 2023-12-25T21:07:29.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.535+0000 7f752759e700 1 -- 172.21.15.174:0/4062507105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7528106bf0 con 0x7f75280601e0 2023-12-25T21:07:29.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.535+0000 7f7516ffd700 1 -- 172.21.15.174:0/4062507105 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7518014d90 con 0x7f75280601e0 2023-12-25T21:07:29.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.536+0000 7f752759e700 1 -- 172.21.15.174:0/4062507105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7528107080 con 0x7f75280601e0 2023-12-25T21:07:29.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.536+0000 7f7516ffd700 1 -- 172.21.15.174:0/4062507105 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75180263a0 con 0x7f75280601e0 2023-12-25T21:07:29.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.536+0000 7f7516ffd700 1 --2- 172.21.15.174:0/4062507105 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f751002ff60 0x7f7510032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:29.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.536+0000 7f7516ffd700 1 -- 172.21.15.174:0/4062507105 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f75180410e0 con 0x7f75280601e0 2023-12-25T21:07:29.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.537+0000 7f752759e700 1 -- 172.21.15.174:0/4062507105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7528045ad0 con 0x7f75280601e0 2023-12-25T21:07:29.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.537+0000 7f7525d9b700 1 --2- 172.21.15.174:0/4062507105 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f751002ff60 0x7f7510032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:29.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.537+0000 7f7525d9b700 1 --2- 172.21.15.174:0/4062507105 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f751002ff60 0x7f7510032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f751c0096a0 tx=0x7f751c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:29.542 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.542+0000 7f7516ffd700 1 -- 172.21.15.174:0/4062507105 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7518012920 con 0x7f75280601e0 2023-12-25T21:07:29.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:29 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:27.970736+0000 mgr.smithi061.jxxeke (mgr.14164) 17 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi061:172.21.15.61=smithi061;smithi174:172.21.15.174=smithi174", "target": ["mon-mgr", ""]}]: dispatch 2023-12-25T21:07:29.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:29 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cephadm 2023-12-25T21:07:27.972183+0000 mgr.smithi061.jxxeke (mgr.14164) 18 : cephadm [INF] Saving service mon spec with placement smithi061:172.21.15.61=smithi061;smithi174:172.21.15.174=smithi174;count:2 2023-12-25T21:07:29.750 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.749+0000 7f752759e700 1 -- 172.21.15.174:0/4062507105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7528107da0 con 0x7f75280601e0 2023-12-25T21:07:29.751 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.751+0000 7f7516ffd700 1 -- 172.21.15.174:0/4062507105 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f751802b020 con 0x7f75280601e0 2023-12-25T21:07:29.751 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:07:29.751 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:07:29.752 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.752+0000 7f752759e700 1 -- 172.21.15.174:0/4062507105 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f751002ff60 msgr2=0x7f7510032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:29.752 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.752+0000 7f752759e700 1 --2- 172.21.15.174:0/4062507105 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f751002ff60 0x7f7510032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f751c0096a0 tx=0x7f751c006b40).stop 2023-12-25T21:07:29.753 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.752+0000 7f752759e700 1 -- 172.21.15.174:0/4062507105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75280601e0 msgr2=0x7f7528106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:29.753 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.752+0000 7f752759e700 1 --2- 172.21.15.174:0/4062507105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75280601e0 0x7f7528106500 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f7518013510 tx=0x7f751800b040).stop 2023-12-25T21:07:29.753 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.752+0000 7f752759e700 1 -- 172.21.15.174:0/4062507105 shutdown_connections 2023-12-25T21:07:29.753 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.752+0000 7f752759e700 1 --2- 172.21.15.174:0/4062507105 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f751002ff60 0x7f7510032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:29.753 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.752+0000 7f752759e700 1 --2- 172.21.15.174:0/4062507105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75280601e0 0x7f7528106500 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:29.753 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.752+0000 7f752759e700 1 -- 172.21.15.174:0/4062507105 >> 172.21.15.174:0/4062507105 conn(0x7f75280f09a0 msgr2=0x7f75280f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:29.753 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.752+0000 7f752759e700 1 -- 172.21.15.174:0/4062507105 shutdown_connections 2023-12-25T21:07:29.753 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:29.752+0000 7f752759e700 1 -- 172.21.15.174:0/4062507105 wait complete. 2023-12-25T21:07:29.753 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:07:30.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:30 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:28.569847+0000 mgr.smithi061.jxxeke (mgr.14164) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:30.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:30 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:29.751366+0000 mon.smithi061 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.174:0/4062507105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:07:31.131 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:07:31.131 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:07:32.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.284+0000 7ff11a0a5700 1 -- 172.21.15.174:0/1144283148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1140f6510 msgr2=0x7ff1140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:32.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.284+0000 7ff11a0a5700 1 --2- 172.21.15.174:0/1144283148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1140f6510 0x7ff1140f6930 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7ff108009fb0 tx=0x7ff108009b30).stop 2023-12-25T21:07:32.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.284+0000 7ff11a0a5700 1 -- 172.21.15.174:0/1144283148 shutdown_connections 2023-12-25T21:07:32.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.284+0000 7ff11a0a5700 1 --2- 172.21.15.174:0/1144283148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1140f6510 0x7ff1140f6930 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:32.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.284+0000 7ff11a0a5700 1 -- 172.21.15.174:0/1144283148 >> 172.21.15.174:0/1144283148 conn(0x7ff1140f1c70 msgr2=0x7ff1140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:32.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.284+0000 7ff11a0a5700 1 -- 172.21.15.174:0/1144283148 shutdown_connections 2023-12-25T21:07:32.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.284+0000 7ff11a0a5700 1 -- 172.21.15.174:0/1144283148 wait complete. 2023-12-25T21:07:32.286 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.285+0000 7ff11a0a5700 1 Processor -- start 2023-12-25T21:07:32.286 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.285+0000 7ff11a0a5700 1 -- start start 2023-12-25T21:07:32.286 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.285+0000 7ff11a0a5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1140f6510 0x7ff114106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:32.286 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.285+0000 7ff11a0a5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff114106c20 con 0x7ff1140f6510 2023-12-25T21:07:32.286 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.286+0000 7ff1137fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1140f6510 0x7ff114106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:32.286 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.286+0000 7ff1137fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1140f6510 0x7ff114106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:39476/0 (socket says 172.21.15.174:39476) 2023-12-25T21:07:32.286 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.286+0000 7ff1137fe700 1 -- 172.21.15.174:0/1941363239 learned_addr learned my addr 172.21.15.174:0/1941363239 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:07:32.286 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.286+0000 7ff1137fe700 1 -- 172.21.15.174:0/1941363239 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff108005040 con 0x7ff1140f6510 2023-12-25T21:07:32.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.286+0000 7ff1137fe700 1 --2- 172.21.15.174:0/1941363239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1140f6510 0x7ff114106730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7ff108013510 tx=0x7ff10800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:32.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.286+0000 7ff0fbfff700 1 -- 172.21.15.174:0/1941363239 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff108009320 con 0x7ff1140f6510 2023-12-25T21:07:32.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.286+0000 7ff11a0a5700 1 -- 172.21.15.174:0/1941363239 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff114106e20 con 0x7ff1140f6510 2023-12-25T21:07:32.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.286+0000 7ff0fbfff700 1 -- 172.21.15.174:0/1941363239 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff108013f00 con 0x7ff1140f6510 2023-12-25T21:07:32.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.286+0000 7ff0fbfff700 1 -- 172.21.15.174:0/1941363239 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff108014d90 con 0x7ff1140f6510 2023-12-25T21:07:32.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.286+0000 7ff11a0a5700 1 -- 172.21.15.174:0/1941363239 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1141072b0 con 0x7ff1140f6510 2023-12-25T21:07:32.288 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.287+0000 7ff0fbfff700 1 -- 172.21.15.174:0/1941363239 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff108014420 con 0x7ff1140f6510 2023-12-25T21:07:32.288 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.287+0000 7ff11a0a5700 1 -- 172.21.15.174:0/1941363239 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff114045ad0 con 0x7ff1140f6510 2023-12-25T21:07:32.288 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.287+0000 7ff0fbfff700 1 --2- 172.21.15.174:0/1941363239 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff0fc02ff60 0x7ff0fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:32.288 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.287+0000 7ff0fbfff700 1 -- 172.21.15.174:0/1941363239 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff1080410b0 con 0x7ff1140f6510 2023-12-25T21:07:32.288 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.287+0000 7ff112ffd700 1 --2- 172.21.15.174:0/1941363239 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff0fc02ff60 0x7ff0fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:32.288 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.288+0000 7ff112ffd700 1 --2- 172.21.15.174:0/1941363239 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff0fc02ff60 0x7ff0fc032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7ff1040096a0 tx=0x7ff104006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:32.292 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.292+0000 7ff0fbfff700 1 -- 172.21.15.174:0/1941363239 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff108026bf0 con 0x7ff1140f6510 2023-12-25T21:07:32.498 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.499+0000 7ff11a0a5700 1 -- 172.21.15.174:0/1941363239 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1140fbf60 con 0x7ff1140f6510 2023-12-25T21:07:32.499 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.499+0000 7ff0fbfff700 1 -- 172.21.15.174:0/1941363239 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff10801db70 con 0x7ff1140f6510 2023-12-25T21:07:32.499 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:07:32.499 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:07:32.501 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.501+0000 7ff11a0a5700 1 -- 172.21.15.174:0/1941363239 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff0fc02ff60 msgr2=0x7ff0fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:32.501 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.501+0000 7ff11a0a5700 1 --2- 172.21.15.174:0/1941363239 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff0fc02ff60 0x7ff0fc032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7ff1040096a0 tx=0x7ff104006b40).stop 2023-12-25T21:07:32.501 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.501+0000 7ff11a0a5700 1 -- 172.21.15.174:0/1941363239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1140f6510 msgr2=0x7ff114106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:32.501 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.501+0000 7ff11a0a5700 1 --2- 172.21.15.174:0/1941363239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1140f6510 0x7ff114106730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7ff108013510 tx=0x7ff10800b040).stop 2023-12-25T21:07:32.501 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.501+0000 7ff11a0a5700 1 -- 172.21.15.174:0/1941363239 shutdown_connections 2023-12-25T21:07:32.501 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.501+0000 7ff11a0a5700 1 --2- 172.21.15.174:0/1941363239 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff0fc02ff60 0x7ff0fc032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:32.501 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.501+0000 7ff11a0a5700 1 --2- 172.21.15.174:0/1941363239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1140f6510 0x7ff114106730 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:32.501 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.501+0000 7ff11a0a5700 1 -- 172.21.15.174:0/1941363239 >> 172.21.15.174:0/1941363239 conn(0x7ff1140f1c70 msgr2=0x7ff1140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:32.501 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.501+0000 7ff11a0a5700 1 -- 172.21.15.174:0/1941363239 shutdown_connections 2023-12-25T21:07:32.502 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:32.501+0000 7ff11a0a5700 1 -- 172.21.15.174:0/1941363239 wait complete. 2023-12-25T21:07:32.502 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:07:32.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:32 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:30.570115+0000 mgr.smithi061.jxxeke (mgr.14164) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:33.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:33 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:32.499865+0000 mon.smithi061 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.174:0/1941363239' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:07:33.917 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:07:33.917 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:07:34.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:34 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:32.570330+0000 mgr.smithi061.jxxeke (mgr.14164) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:35.080 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.080+0000 7f88b357f700 1 -- 172.21.15.174:0/4101473838 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88ac0f51a0 msgr2=0x7f88ac0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:35.081 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.080+0000 7f88b357f700 1 --2- 172.21.15.174:0/4101473838 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88ac0f51a0 0x7f88ac0f55c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f889c009fb0 tx=0x7f889c009b30).stop 2023-12-25T21:07:35.081 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.080+0000 7f88b357f700 1 -- 172.21.15.174:0/4101473838 shutdown_connections 2023-12-25T21:07:35.081 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.080+0000 7f88b357f700 1 --2- 172.21.15.174:0/4101473838 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88ac0f51a0 0x7f88ac0f55c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:35.081 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.080+0000 7f88b357f700 1 -- 172.21.15.174:0/4101473838 >> 172.21.15.174:0/4101473838 conn(0x7f88ac0f09a0 msgr2=0x7f88ac0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:35.081 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.080+0000 7f88b357f700 1 -- 172.21.15.174:0/4101473838 shutdown_connections 2023-12-25T21:07:35.081 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.080+0000 7f88b357f700 1 -- 172.21.15.174:0/4101473838 wait complete. 2023-12-25T21:07:35.081 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.080+0000 7f88b357f700 1 Processor -- start 2023-12-25T21:07:35.081 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.080+0000 7f88b357f700 1 -- start start 2023-12-25T21:07:35.081 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.080+0000 7f88b357f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88ac0f51a0 0x7f88ac0fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:35.081 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.080+0000 7f88b357f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac0fb5a0 con 0x7f88ac0f51a0 2023-12-25T21:07:35.081 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.081+0000 7f88b257d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88ac0f51a0 0x7f88ac0fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:35.082 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.081+0000 7f88b257d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88ac0f51a0 0x7f88ac0fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:39482/0 (socket says 172.21.15.174:39482) 2023-12-25T21:07:35.082 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.081+0000 7f88b257d700 1 -- 172.21.15.174:0/3895681858 learned_addr learned my addr 172.21.15.174:0/3895681858 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:07:35.082 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.081+0000 7f88b257d700 1 -- 172.21.15.174:0/3895681858 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c005040 con 0x7f88ac0f51a0 2023-12-25T21:07:35.082 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.081+0000 7f88b257d700 1 --2- 172.21.15.174:0/3895681858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88ac0f51a0 0x7f88ac0fd120 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f889c013510 tx=0x7f889c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:35.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.081+0000 7f88a2ffd700 1 -- 172.21.15.174:0/3895681858 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f889c009320 con 0x7f88ac0f51a0 2023-12-25T21:07:35.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.081+0000 7f88b357f700 1 -- 172.21.15.174:0/3895681858 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88ac0fb740 con 0x7f88ac0f51a0 2023-12-25T21:07:35.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.081+0000 7f88a2ffd700 1 -- 172.21.15.174:0/3895681858 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f889c013f00 con 0x7f88ac0f51a0 2023-12-25T21:07:35.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.081+0000 7f88a2ffd700 1 -- 172.21.15.174:0/3895681858 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f889c014dc0 con 0x7f88ac0f51a0 2023-12-25T21:07:35.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.081+0000 7f88b357f700 1 -- 172.21.15.174:0/3895681858 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88ac0fbbd0 con 0x7f88ac0f51a0 2023-12-25T21:07:35.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.082+0000 7f88a2ffd700 1 -- 172.21.15.174:0/3895681858 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f889c014420 con 0x7f88ac0f51a0 2023-12-25T21:07:35.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.082+0000 7f88b357f700 1 -- 172.21.15.174:0/3895681858 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88ac059100 con 0x7f88ac0f51a0 2023-12-25T21:07:35.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.082+0000 7f88a2ffd700 1 --2- 172.21.15.174:0/3895681858 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f889802ff60 0x7f8898032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:35.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.082+0000 7f88a2ffd700 1 -- 172.21.15.174:0/3895681858 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f889c0410e0 con 0x7f88ac0f51a0 2023-12-25T21:07:35.084 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.083+0000 7f88b1d7c700 1 --2- 172.21.15.174:0/3895681858 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f889802ff60 0x7f8898032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:35.084 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.083+0000 7f88b1d7c700 1 --2- 172.21.15.174:0/3895681858 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f889802ff60 0x7f8898032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f88a80096a0 tx=0x7f88a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:35.087 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.087+0000 7f88a2ffd700 1 -- 172.21.15.174:0/3895681858 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f889c0095f0 con 0x7f88ac0f51a0 2023-12-25T21:07:35.292 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.292+0000 7f88b357f700 1 -- 172.21.15.174:0/3895681858 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88ac045ad0 con 0x7f88ac0f51a0 2023-12-25T21:07:35.292 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.293+0000 7f88a2ffd700 1 -- 172.21.15.174:0/3895681858 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f889c01cb10 con 0x7f88ac0f51a0 2023-12-25T21:07:35.293 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:07:35.293 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:07:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.294+0000 7f88b357f700 1 -- 172.21.15.174:0/3895681858 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f889802ff60 msgr2=0x7f8898032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.294+0000 7f88b357f700 1 --2- 172.21.15.174:0/3895681858 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f889802ff60 0x7f8898032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f88a80096a0 tx=0x7f88a8006b40).stop 2023-12-25T21:07:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.294+0000 7f88b357f700 1 -- 172.21.15.174:0/3895681858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88ac0f51a0 msgr2=0x7f88ac0fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.294+0000 7f88b357f700 1 --2- 172.21.15.174:0/3895681858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88ac0f51a0 0x7f88ac0fd120 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f889c013510 tx=0x7f889c00b040).stop 2023-12-25T21:07:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.294+0000 7f88b357f700 1 -- 172.21.15.174:0/3895681858 shutdown_connections 2023-12-25T21:07:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.294+0000 7f88b357f700 1 --2- 172.21.15.174:0/3895681858 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f889802ff60 0x7f8898032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.294+0000 7f88b357f700 1 --2- 172.21.15.174:0/3895681858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88ac0f51a0 0x7f88ac0fd120 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.294+0000 7f88b357f700 1 -- 172.21.15.174:0/3895681858 >> 172.21.15.174:0/3895681858 conn(0x7f88ac0f09a0 msgr2=0x7f88ac05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.294+0000 7f88b357f700 1 -- 172.21.15.174:0/3895681858 shutdown_connections 2023-12-25T21:07:35.295 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:35.294+0000 7f88b357f700 1 -- 172.21.15.174:0/3895681858 wait complete. 2023-12-25T21:07:35.295 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:07:35.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:35 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:35.293085+0000 mon.smithi061 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.174:0/3895681858' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:07:36.682 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:07:36.682 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:07:36.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:36 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:34.570639+0000 mgr.smithi061.jxxeke (mgr.14164) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:37.853 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.849+0000 7fdc119de700 1 -- 172.21.15.174:0/183025200 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc0c0f5410 msgr2=0x7fdc0c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:37.853 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.849+0000 7fdc119de700 1 --2- 172.21.15.174:0/183025200 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc0c0f5410 0x7fdc0c0f5830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fdbf4009fb0 tx=0x7fdbf4009b30).stop 2023-12-25T21:07:37.853 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.850+0000 7fdc119de700 1 -- 172.21.15.174:0/183025200 shutdown_connections 2023-12-25T21:07:37.853 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.850+0000 7fdc119de700 1 --2- 172.21.15.174:0/183025200 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc0c0f5410 0x7fdc0c0f5830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:37.853 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.850+0000 7fdc119de700 1 -- 172.21.15.174:0/183025200 >> 172.21.15.174:0/183025200 conn(0x7fdc0c0f0b70 msgr2=0x7fdc0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:37.853 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.850+0000 7fdc119de700 1 -- 172.21.15.174:0/183025200 shutdown_connections 2023-12-25T21:07:37.854 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.850+0000 7fdc119de700 1 -- 172.21.15.174:0/183025200 wait complete. 2023-12-25T21:07:37.854 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.850+0000 7fdc119de700 1 Processor -- start 2023-12-25T21:07:37.854 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.850+0000 7fdc119de700 1 -- start start 2023-12-25T21:07:37.854 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.850+0000 7fdc119de700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc0c0f5410 0x7fdc0c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:37.854 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.850+0000 7fdc119de700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c0fb820 con 0x7fdc0c0f5410 2023-12-25T21:07:37.854 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.851+0000 7fdc0affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc0c0f5410 0x7fdc0c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:37.855 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.851+0000 7fdc0affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc0c0f5410 0x7fdc0c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:45856/0 (socket says 172.21.15.174:45856) 2023-12-25T21:07:37.855 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.851+0000 7fdc0affd700 1 -- 172.21.15.174:0/891984180 learned_addr learned my addr 172.21.15.174:0/891984180 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:07:37.855 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.851+0000 7fdc0affd700 1 -- 172.21.15.174:0/891984180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf4005040 con 0x7fdc0c0f5410 2023-12-25T21:07:37.855 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.851+0000 7fdc0affd700 1 --2- 172.21.15.174:0/891984180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc0c0f5410 0x7fdc0c0fd3a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fdbf4009f80 tx=0x7fdbf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:37.855 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.851+0000 7fdc037fe700 1 -- 172.21.15.174:0/891984180 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbf4009320 con 0x7fdc0c0f5410 2023-12-25T21:07:37.855 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.851+0000 7fdc119de700 1 -- 172.21.15.174:0/891984180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc0c0fb9c0 con 0x7fdc0c0f5410 2023-12-25T21:07:37.856 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.851+0000 7fdc037fe700 1 -- 172.21.15.174:0/891984180 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdbf4013ec0 con 0x7fdc0c0f5410 2023-12-25T21:07:37.856 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.851+0000 7fdc119de700 1 -- 172.21.15.174:0/891984180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc0c0fbe50 con 0x7fdc0c0f5410 2023-12-25T21:07:37.856 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.851+0000 7fdc037fe700 1 -- 172.21.15.174:0/891984180 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbf401ece0 con 0x7fdc0c0f5410 2023-12-25T21:07:37.856 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.852+0000 7fdc119de700 1 -- 172.21.15.174:0/891984180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc0c059100 con 0x7fdc0c0f5410 2023-12-25T21:07:37.856 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.852+0000 7fdc037fe700 1 -- 172.21.15.174:0/891984180 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdbf401e420 con 0x7fdc0c0f5410 2023-12-25T21:07:37.856 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.852+0000 7fdc037fe700 1 --2- 172.21.15.174:0/891984180 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdbf802ff60 0x7fdbf8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:37.857 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.852+0000 7fdc037fe700 1 -- 172.21.15.174:0/891984180 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdbf4040e70 con 0x7fdc0c0f5410 2023-12-25T21:07:37.857 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.853+0000 7fdc0a7fc700 1 --2- 172.21.15.174:0/891984180 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdbf802ff60 0x7fdbf8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:37.857 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.853+0000 7fdc0a7fc700 1 --2- 172.21.15.174:0/891984180 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdbf802ff60 0x7fdbf8032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fdbfc0096a0 tx=0x7fdbfc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:37.858 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:37.857+0000 7fdc037fe700 1 -- 172.21.15.174:0/891984180 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdbf401e780 con 0x7fdc0c0f5410 2023-12-25T21:07:38.063 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:38.064+0000 7fdc119de700 1 -- 172.21.15.174:0/891984180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc0c0fae60 con 0x7fdc0c0f5410 2023-12-25T21:07:38.064 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:38.065+0000 7fdc037fe700 1 -- 172.21.15.174:0/891984180 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdbf401cb10 con 0x7fdc0c0f5410 2023-12-25T21:07:38.064 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:07:38.064 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:07:38.069 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:38.066+0000 7fdc119de700 1 -- 172.21.15.174:0/891984180 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdbf802ff60 msgr2=0x7fdbf8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:38.069 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:38.066+0000 7fdc119de700 1 --2- 172.21.15.174:0/891984180 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdbf802ff60 0x7fdbf8032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fdbfc0096a0 tx=0x7fdbfc006b40).stop 2023-12-25T21:07:38.069 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:38.066+0000 7fdc119de700 1 -- 172.21.15.174:0/891984180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc0c0f5410 msgr2=0x7fdc0c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:38.070 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:38.066+0000 7fdc119de700 1 --2- 172.21.15.174:0/891984180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc0c0f5410 0x7fdc0c0fd3a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fdbf4009f80 tx=0x7fdbf400b040).stop 2023-12-25T21:07:38.070 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:38.066+0000 7fdc119de700 1 -- 172.21.15.174:0/891984180 shutdown_connections 2023-12-25T21:07:38.070 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:38.066+0000 7fdc119de700 1 --2- 172.21.15.174:0/891984180 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdbf802ff60 0x7fdbf8032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:38.070 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:38.066+0000 7fdc119de700 1 --2- 172.21.15.174:0/891984180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc0c0f5410 0x7fdc0c0fd3a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:38.070 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:38.066+0000 7fdc119de700 1 -- 172.21.15.174:0/891984180 >> 172.21.15.174:0/891984180 conn(0x7fdc0c0f0b70 msgr2=0x7fdc0c05fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:38.070 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:38.066+0000 7fdc119de700 1 -- 172.21.15.174:0/891984180 shutdown_connections 2023-12-25T21:07:38.070 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:38.066+0000 7fdc119de700 1 -- 172.21.15.174:0/891984180 wait complete. 2023-12-25T21:07:38.070 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:07:38.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:38 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:36.571079+0000 mgr.smithi061.jxxeke (mgr.14164) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:38.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:38 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:38.064819+0000 mon.smithi061 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.174:0/891984180' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:07:39.443 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:07:39.444 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:07:40.633 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.632+0000 7fdeed077700 1 -- 172.21.15.174:0/3788216823 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdee80f5410 msgr2=0x7fdee80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.632+0000 7fdeed077700 1 --2- 172.21.15.174:0/3788216823 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdee80f5410 0x7fdee80f5830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fdedc009fb0 tx=0x7fdedc009b30).stop 2023-12-25T21:07:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.632+0000 7fdeed077700 1 -- 172.21.15.174:0/3788216823 shutdown_connections 2023-12-25T21:07:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.632+0000 7fdeed077700 1 --2- 172.21.15.174:0/3788216823 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdee80f5410 0x7fdee80f5830 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.632+0000 7fdeed077700 1 -- 172.21.15.174:0/3788216823 >> 172.21.15.174:0/3788216823 conn(0x7fdee80f0b70 msgr2=0x7fdee80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.632+0000 7fdeed077700 1 -- 172.21.15.174:0/3788216823 shutdown_connections 2023-12-25T21:07:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.632+0000 7fdeed077700 1 -- 172.21.15.174:0/3788216823 wait complete. 2023-12-25T21:07:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.632+0000 7fdeed077700 1 Processor -- start 2023-12-25T21:07:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.633+0000 7fdeed077700 1 -- start start 2023-12-25T21:07:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.633+0000 7fdeed077700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdee8105740 0x7fdee8105b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.633+0000 7fdeed077700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee8106050 con 0x7fdee8105740 2023-12-25T21:07:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.633+0000 7fdee6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdee8105740 0x7fdee8105b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.633+0000 7fdee6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdee8105740 0x7fdee8105b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:45870/0 (socket says 172.21.15.174:45870) 2023-12-25T21:07:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.633+0000 7fdee6d9d700 1 -- 172.21.15.174:0/2191069605 learned_addr learned my addr 172.21.15.174:0/2191069605 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:07:40.635 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.633+0000 7fdee6d9d700 1 -- 172.21.15.174:0/2191069605 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdedc005040 con 0x7fdee8105740 2023-12-25T21:07:40.635 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.633+0000 7fdee6d9d700 1 --2- 172.21.15.174:0/2191069605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdee8105740 0x7fdee8105b60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fdedc009f80 tx=0x7fdedc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:40.635 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.634+0000 7fdecf7fe700 1 -- 172.21.15.174:0/2191069605 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdedc0136c0 con 0x7fdee8105740 2023-12-25T21:07:40.635 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.634+0000 7fdeed077700 1 -- 172.21.15.174:0/2191069605 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee8106250 con 0x7fdee8105740 2023-12-25T21:07:40.635 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.634+0000 7fdecf7fe700 1 -- 172.21.15.174:0/2191069605 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdedc0095f0 con 0x7fdee8105740 2023-12-25T21:07:40.635 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.634+0000 7fdecf7fe700 1 -- 172.21.15.174:0/2191069605 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdedc014420 con 0x7fdee8105740 2023-12-25T21:07:40.635 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.634+0000 7fdeed077700 1 -- 172.21.15.174:0/2191069605 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee8106da0 con 0x7fdee8105740 2023-12-25T21:07:40.636 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.634+0000 7fdecf7fe700 1 -- 172.21.15.174:0/2191069605 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdedc025060 con 0x7fdee8105740 2023-12-25T21:07:40.636 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.634+0000 7fdeed077700 1 -- 172.21.15.174:0/2191069605 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee8059100 con 0x7fdee8105740 2023-12-25T21:07:40.636 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.634+0000 7fdecf7fe700 1 --2- 172.21.15.174:0/2191069605 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fded002ff60 0x7fded0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:40.636 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.635+0000 7fdecf7fe700 1 -- 172.21.15.174:0/2191069605 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdedc040da0 con 0x7fdee8105740 2023-12-25T21:07:40.636 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.635+0000 7fdee659c700 1 --2- 172.21.15.174:0/2191069605 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fded002ff60 0x7fded0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:40.636 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.635+0000 7fdee659c700 1 --2- 172.21.15.174:0/2191069605 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fded002ff60 0x7fded0032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fded80096a0 tx=0x7fded8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:40.639 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.639+0000 7fdecf7fe700 1 -- 172.21.15.174:0/2191069605 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdedc01c950 con 0x7fdee8105740 2023-12-25T21:07:40.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:40 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:38.571484+0000 mgr.smithi061.jxxeke (mgr.14164) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:40.844 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.845+0000 7fdeed077700 1 -- 172.21.15.174:0/2191069605 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdee8107560 con 0x7fdee8105740 2023-12-25T21:07:40.845 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.845+0000 7fdecf7fe700 1 -- 172.21.15.174:0/2191069605 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdedc01cb60 con 0x7fdee8105740 2023-12-25T21:07:40.845 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:07:40.845 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:07:40.847 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.847+0000 7fdeed077700 1 -- 172.21.15.174:0/2191069605 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fded002ff60 msgr2=0x7fded0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:40.847 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.847+0000 7fdeed077700 1 --2- 172.21.15.174:0/2191069605 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fded002ff60 0x7fded0032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fded80096a0 tx=0x7fded8006b40).stop 2023-12-25T21:07:40.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.847+0000 7fdeed077700 1 -- 172.21.15.174:0/2191069605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdee8105740 msgr2=0x7fdee8105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:40.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.847+0000 7fdeed077700 1 --2- 172.21.15.174:0/2191069605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdee8105740 0x7fdee8105b60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fdedc009f80 tx=0x7fdedc009b30).stop 2023-12-25T21:07:40.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.847+0000 7fdeed077700 1 -- 172.21.15.174:0/2191069605 shutdown_connections 2023-12-25T21:07:40.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.847+0000 7fdeed077700 1 --2- 172.21.15.174:0/2191069605 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fded002ff60 0x7fded0032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:40.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.847+0000 7fdeed077700 1 --2- 172.21.15.174:0/2191069605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdee8105740 0x7fdee8105b60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:40.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.847+0000 7fdeed077700 1 -- 172.21.15.174:0/2191069605 >> 172.21.15.174:0/2191069605 conn(0x7fdee80f0b70 msgr2=0x7fdee80f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:40.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.847+0000 7fdeed077700 1 -- 172.21.15.174:0/2191069605 shutdown_connections 2023-12-25T21:07:40.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:40.847+0000 7fdeed077700 1 -- 172.21.15.174:0/2191069605 wait complete. 2023-12-25T21:07:40.848 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:07:41.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:41 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:40.845797+0000 mon.smithi061 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.174:0/2191069605' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:07:42.188 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:07:42.188 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:07:42.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:42 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:40.571789+0000 mgr.smithi061.jxxeke (mgr.14164) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:43.376 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.375+0000 7f1d240a0700 1 -- 172.21.15.174:0/1105259124 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d1c0f5410 msgr2=0x7f1d1c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:43.376 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.375+0000 7f1d240a0700 1 --2- 172.21.15.174:0/1105259124 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d1c0f5410 0x7f1d1c0f5830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f1d0c009fb0 tx=0x7f1d0c009b30).stop 2023-12-25T21:07:43.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.375+0000 7f1d240a0700 1 -- 172.21.15.174:0/1105259124 shutdown_connections 2023-12-25T21:07:43.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.375+0000 7f1d240a0700 1 --2- 172.21.15.174:0/1105259124 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d1c0f5410 0x7f1d1c0f5830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:43.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.375+0000 7f1d240a0700 1 -- 172.21.15.174:0/1105259124 >> 172.21.15.174:0/1105259124 conn(0x7f1d1c0f0b70 msgr2=0x7f1d1c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:43.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.375+0000 7f1d240a0700 1 -- 172.21.15.174:0/1105259124 shutdown_connections 2023-12-25T21:07:43.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.375+0000 7f1d240a0700 1 -- 172.21.15.174:0/1105259124 wait complete. 2023-12-25T21:07:43.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.376+0000 7f1d240a0700 1 Processor -- start 2023-12-25T21:07:43.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.376+0000 7f1d240a0700 1 -- start start 2023-12-25T21:07:43.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.376+0000 7f1d240a0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d1c0f5410 0x7f1d1c1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:43.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.376+0000 7f1d240a0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c060280 con 0x7f1d1c0f5410 2023-12-25T21:07:43.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.376+0000 7f1d21e3c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d1c0f5410 0x7f1d1c1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:43.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.376+0000 7f1d21e3c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d1c0f5410 0x7f1d1c1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:45882/0 (socket says 172.21.15.174:45882) 2023-12-25T21:07:43.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.376+0000 7f1d21e3c700 1 -- 172.21.15.174:0/2025006136 learned_addr learned my addr 172.21.15.174:0/2025006136 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:07:43.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.376+0000 7f1d21e3c700 1 -- 172.21.15.174:0/2025006136 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0c005040 con 0x7f1d1c0f5410 2023-12-25T21:07:43.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.376+0000 7f1d21e3c700 1 --2- 172.21.15.174:0/2025006136 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d1c0f5410 0x7f1d1c1093f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f1d0c014000 tx=0x7f1d0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:43.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.377+0000 7f1d127fc700 1 -- 172.21.15.174:0/2025006136 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d0c0095f0 con 0x7f1d1c0f5410 2023-12-25T21:07:43.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.377+0000 7f1d240a0700 1 -- 172.21.15.174:0/2025006136 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d1c060480 con 0x7f1d1c0f5410 2023-12-25T21:07:43.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.377+0000 7f1d127fc700 1 -- 172.21.15.174:0/2025006136 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d0c007e70 con 0x7f1d1c0f5410 2023-12-25T21:07:43.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.377+0000 7f1d127fc700 1 -- 172.21.15.174:0/2025006136 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d0c014ea0 con 0x7f1d1c0f5410 2023-12-25T21:07:43.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.377+0000 7f1d240a0700 1 -- 172.21.15.174:0/2025006136 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1c109b20 con 0x7f1d1c0f5410 2023-12-25T21:07:43.379 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.377+0000 7f1d127fc700 1 -- 172.21.15.174:0/2025006136 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d0c01d4e0 con 0x7f1d1c0f5410 2023-12-25T21:07:43.379 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.378+0000 7f1d127fc700 1 --2- 172.21.15.174:0/2025006136 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1d0802ff60 0x7f1d08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:43.379 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.378+0000 7f1d127fc700 1 -- 172.21.15.174:0/2025006136 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d0c040ca0 con 0x7f1d1c0f5410 2023-12-25T21:07:43.379 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.378+0000 7f1d240a0700 1 -- 172.21.15.174:0/2025006136 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d1c045ad0 con 0x7f1d1c0f5410 2023-12-25T21:07:43.379 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.378+0000 7f1d2163b700 1 --2- 172.21.15.174:0/2025006136 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1d0802ff60 0x7f1d08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:43.379 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.378+0000 7f1d2163b700 1 --2- 172.21.15.174:0/2025006136 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1d0802ff60 0x7f1d08032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f1d180096a0 tx=0x7f1d18006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:43.383 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.383+0000 7f1d127fc700 1 -- 172.21.15.174:0/2025006136 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d0c00f6c0 con 0x7f1d1c0f5410 2023-12-25T21:07:43.589 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.589+0000 7f1d240a0700 1 -- 172.21.15.174:0/2025006136 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d1c0fae60 con 0x7f1d1c0f5410 2023-12-25T21:07:43.590 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.590+0000 7f1d127fc700 1 -- 172.21.15.174:0/2025006136 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1d1c0fae60 con 0x7f1d1c0f5410 2023-12-25T21:07:43.590 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:07:43.590 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:07:43.591 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.592+0000 7f1d240a0700 1 -- 172.21.15.174:0/2025006136 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1d0802ff60 msgr2=0x7f1d08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:43.592 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.592+0000 7f1d240a0700 1 --2- 172.21.15.174:0/2025006136 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1d0802ff60 0x7f1d08032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f1d180096a0 tx=0x7f1d18006b40).stop 2023-12-25T21:07:43.592 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.592+0000 7f1d240a0700 1 -- 172.21.15.174:0/2025006136 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d1c0f5410 msgr2=0x7f1d1c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:43.592 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.592+0000 7f1d240a0700 1 --2- 172.21.15.174:0/2025006136 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d1c0f5410 0x7f1d1c1093f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f1d0c014000 tx=0x7f1d0c00b040).stop 2023-12-25T21:07:43.592 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.592+0000 7f1d240a0700 1 -- 172.21.15.174:0/2025006136 shutdown_connections 2023-12-25T21:07:43.592 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.592+0000 7f1d240a0700 1 --2- 172.21.15.174:0/2025006136 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1d0802ff60 0x7f1d08032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:43.592 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.592+0000 7f1d240a0700 1 --2- 172.21.15.174:0/2025006136 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d1c0f5410 0x7f1d1c1093f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:43.592 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.592+0000 7f1d240a0700 1 -- 172.21.15.174:0/2025006136 >> 172.21.15.174:0/2025006136 conn(0x7f1d1c0f0b70 msgr2=0x7f1d1c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:43.592 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.592+0000 7f1d240a0700 1 -- 172.21.15.174:0/2025006136 shutdown_connections 2023-12-25T21:07:43.592 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:43.592+0000 7f1d240a0700 1 -- 172.21.15.174:0/2025006136 wait complete. 2023-12-25T21:07:43.592 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:07:44.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:44 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:42.572052+0000 mgr.smithi061.jxxeke (mgr.14164) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:44.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:44 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:43.590517+0000 mon.smithi061 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.174:0/2025006136' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:07:44.986 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:07:44.986 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:07:46.160 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.159+0000 7f2bc9236700 1 -- 172.21.15.174:0/2385108457 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc40f5410 msgr2=0x7f2bc40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:46.160 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.159+0000 7f2bc9236700 1 --2- 172.21.15.174:0/2385108457 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc40f5410 0x7f2bc40f5830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f2bb8009fb0 tx=0x7f2bb8009b30).stop 2023-12-25T21:07:46.160 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.159+0000 7f2bc9236700 1 -- 172.21.15.174:0/2385108457 shutdown_connections 2023-12-25T21:07:46.160 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.159+0000 7f2bc9236700 1 --2- 172.21.15.174:0/2385108457 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc40f5410 0x7f2bc40f5830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:46.160 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.159+0000 7f2bc9236700 1 -- 172.21.15.174:0/2385108457 >> 172.21.15.174:0/2385108457 conn(0x7f2bc40f0b70 msgr2=0x7f2bc40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:46.160 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.159+0000 7f2bc9236700 1 -- 172.21.15.174:0/2385108457 shutdown_connections 2023-12-25T21:07:46.160 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.159+0000 7f2bc9236700 1 -- 172.21.15.174:0/2385108457 wait complete. 2023-12-25T21:07:46.160 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.160+0000 7f2bc9236700 1 Processor -- start 2023-12-25T21:07:46.160 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.160+0000 7f2bc9236700 1 -- start start 2023-12-25T21:07:46.160 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.160+0000 7f2bc9236700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc40f5410 0x7f2bc40fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:46.160 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.160+0000 7f2bc9236700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc40fb820 con 0x7f2bc40f5410 2023-12-25T21:07:46.160 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.160+0000 7f2bc2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc40f5410 0x7f2bc40fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:46.161 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.160+0000 7f2bc2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc40f5410 0x7f2bc40fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:45896/0 (socket says 172.21.15.174:45896) 2023-12-25T21:07:46.161 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.160+0000 7f2bc2d9d700 1 -- 172.21.15.174:0/3658994291 learned_addr learned my addr 172.21.15.174:0/3658994291 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:07:46.165 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.160+0000 7f2bc2d9d700 1 -- 172.21.15.174:0/3658994291 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb8005040 con 0x7f2bc40f5410 2023-12-25T21:07:46.165 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.160+0000 7f2bc2d9d700 1 --2- 172.21.15.174:0/3658994291 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc40f5410 0x7f2bc40fd3a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f2bb8013510 tx=0x7f2bb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:46.165 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.161+0000 7f2bab7fe700 1 -- 172.21.15.174:0/3658994291 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb8009320 con 0x7f2bc40f5410 2023-12-25T21:07:46.165 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.161+0000 7f2bc9236700 1 -- 172.21.15.174:0/3658994291 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc40fb9c0 con 0x7f2bc40f5410 2023-12-25T21:07:46.165 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.161+0000 7f2bab7fe700 1 -- 172.21.15.174:0/3658994291 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2bb8013f00 con 0x7f2bc40f5410 2023-12-25T21:07:46.165 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.161+0000 7f2bab7fe700 1 -- 172.21.15.174:0/3658994291 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb8014d90 con 0x7f2bc40f5410 2023-12-25T21:07:46.165 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.161+0000 7f2bc9236700 1 -- 172.21.15.174:0/3658994291 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc40fbe50 con 0x7f2bc40f5410 2023-12-25T21:07:46.165 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.161+0000 7f2bab7fe700 1 -- 172.21.15.174:0/3658994291 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2bb8014420 con 0x7f2bc40f5410 2023-12-25T21:07:46.166 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.162+0000 7f2bc9236700 1 -- 172.21.15.174:0/3658994291 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc4059100 con 0x7f2bc40f5410 2023-12-25T21:07:46.166 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.162+0000 7f2bab7fe700 1 --2- 172.21.15.174:0/3658994291 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2bac02ffb0 0x7f2bac032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:46.166 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.162+0000 7f2bab7fe700 1 -- 172.21.15.174:0/3658994291 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2bb8041090 con 0x7f2bc40f5410 2023-12-25T21:07:46.166 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.162+0000 7f2bc259c700 1 --2- 172.21.15.174:0/3658994291 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2bac02ffb0 0x7f2bac032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:46.166 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.162+0000 7f2bc259c700 1 --2- 172.21.15.174:0/3658994291 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2bac02ffb0 0x7f2bac032470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f2bb40096a0 tx=0x7f2bb4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:46.166 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.167+0000 7f2bab7fe700 1 -- 172.21.15.174:0/3658994291 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2bb806d2b0 con 0x7f2bc40f5410 2023-12-25T21:07:46.370 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.370+0000 7f2bc9236700 1 -- 172.21.15.174:0/3658994291 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2bc40fae60 con 0x7f2bc40f5410 2023-12-25T21:07:46.370 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.371+0000 7f2bab7fe700 1 -- 172.21.15.174:0/3658994291 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2bb80095f0 con 0x7f2bc40f5410 2023-12-25T21:07:46.371 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:07:46.371 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:07:46.372 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.372+0000 7f2bc9236700 1 -- 172.21.15.174:0/3658994291 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2bac02ffb0 msgr2=0x7f2bac032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:46.372 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.372+0000 7f2bc9236700 1 --2- 172.21.15.174:0/3658994291 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2bac02ffb0 0x7f2bac032470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f2bb40096a0 tx=0x7f2bb4006b40).stop 2023-12-25T21:07:46.372 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.372+0000 7f2bc9236700 1 -- 172.21.15.174:0/3658994291 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc40f5410 msgr2=0x7f2bc40fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:46.372 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.372+0000 7f2bc9236700 1 --2- 172.21.15.174:0/3658994291 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc40f5410 0x7f2bc40fd3a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f2bb8013510 tx=0x7f2bb800b040).stop 2023-12-25T21:07:46.372 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.372+0000 7f2bc9236700 1 -- 172.21.15.174:0/3658994291 shutdown_connections 2023-12-25T21:07:46.373 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.372+0000 7f2bc9236700 1 --2- 172.21.15.174:0/3658994291 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2bac02ffb0 0x7f2bac032470 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:46.373 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.372+0000 7f2bc9236700 1 --2- 172.21.15.174:0/3658994291 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bc40f5410 0x7f2bc40fd3a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:46.373 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.372+0000 7f2bc9236700 1 -- 172.21.15.174:0/3658994291 >> 172.21.15.174:0/3658994291 conn(0x7f2bc40f0b70 msgr2=0x7f2bc405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:46.373 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.372+0000 7f2bc9236700 1 -- 172.21.15.174:0/3658994291 shutdown_connections 2023-12-25T21:07:46.373 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:46.372+0000 7f2bc9236700 1 -- 172.21.15.174:0/3658994291 wait complete. 2023-12-25T21:07:46.373 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:07:46.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:46 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:44.572392+0000 mgr.smithi061.jxxeke (mgr.14164) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:47.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:47 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:46.371092+0000 mon.smithi061 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.174:0/3658994291' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:07:47.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:47 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:46.572678+0000 mgr.smithi061.jxxeke (mgr.14164) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:47.803 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:07:47.803 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:07:49.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:49 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:48.572962+0000 mgr.smithi061.jxxeke (mgr.14164) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:52.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:52 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:50.573392+0000 mgr.smithi061.jxxeke (mgr.14164) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:52.860 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.722+0000 7fe6a8da0700 1 -- 172.21.15.174:0/3284483377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a40f5410 msgr2=0x7fe6a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:52.860 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.722+0000 7fe6a8da0700 1 --2- 172.21.15.174:0/3284483377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a40f5410 0x7fe6a40f5830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fe694009fb0 tx=0x7fe694009b30).stop 2023-12-25T21:07:52.861 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.722+0000 7fe6a8da0700 1 -- 172.21.15.174:0/3284483377 shutdown_connections 2023-12-25T21:07:52.861 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.722+0000 7fe6a8da0700 1 --2- 172.21.15.174:0/3284483377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a40f5410 0x7fe6a40f5830 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:52.861 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.722+0000 7fe6a8da0700 1 -- 172.21.15.174:0/3284483377 >> 172.21.15.174:0/3284483377 conn(0x7fe6a40f0b70 msgr2=0x7fe6a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:52.861 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.722+0000 7fe6a8da0700 1 -- 172.21.15.174:0/3284483377 shutdown_connections 2023-12-25T21:07:52.861 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.723+0000 7fe6a8da0700 1 -- 172.21.15.174:0/3284483377 wait complete. 2023-12-25T21:07:52.861 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.723+0000 7fe6a8da0700 1 Processor -- start 2023-12-25T21:07:52.861 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.723+0000 7fe6a8da0700 1 -- start start 2023-12-25T21:07:52.861 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.723+0000 7fe6a8da0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a40f5410 0x7fe6a4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:52.861 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.723+0000 7fe6a8da0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a4105b20 con 0x7fe6a40f5410 2023-12-25T21:07:52.861 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.724+0000 7fe6a259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a40f5410 0x7fe6a4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:52.861 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.724+0000 7fe6a259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a40f5410 0x7fe6a4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:40828/0 (socket says 172.21.15.174:40828) 2023-12-25T21:07:52.861 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.724+0000 7fe6a259c700 1 -- 172.21.15.174:0/53575053 learned_addr learned my addr 172.21.15.174:0/53575053 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:07:52.862 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.724+0000 7fe6a259c700 1 -- 172.21.15.174:0/53575053 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe694005040 con 0x7fe6a40f5410 2023-12-25T21:07:52.862 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.724+0000 7fe6a259c700 1 --2- 172.21.15.174:0/53575053 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a40f5410 0x7fe6a4105630 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe694013510 tx=0x7fe69400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:52.862 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.724+0000 7fe69affd700 1 -- 172.21.15.174:0/53575053 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe694009320 con 0x7fe6a40f5410 2023-12-25T21:07:52.862 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.724+0000 7fe69affd700 1 -- 172.21.15.174:0/53575053 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe694013f00 con 0x7fe6a40f5410 2023-12-25T21:07:52.862 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.724+0000 7fe69affd700 1 -- 172.21.15.174:0/53575053 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe694014d90 con 0x7fe6a40f5410 2023-12-25T21:07:52.862 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.725+0000 7fe6a8da0700 1 -- 172.21.15.174:0/53575053 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a4105d20 con 0x7fe6a40f5410 2023-12-25T21:07:52.862 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.725+0000 7fe6a8da0700 1 -- 172.21.15.174:0/53575053 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a41061b0 con 0x7fe6a40f5410 2023-12-25T21:07:52.862 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.725+0000 7fe69affd700 1 -- 172.21.15.174:0/53575053 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe694014420 con 0x7fe6a40f5410 2023-12-25T21:07:52.862 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.725+0000 7fe6a8da0700 1 -- 172.21.15.174:0/53575053 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a4045ad0 con 0x7fe6a40f5410 2023-12-25T21:07:52.862 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.726+0000 7fe69affd700 1 --2- 172.21.15.174:0/53575053 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe69002ffb0 0x7fe690032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:52.862 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.726+0000 7fe69affd700 1 -- 172.21.15.174:0/53575053 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe694040ff0 con 0x7fe6a40f5410 2023-12-25T21:07:52.862 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.726+0000 7fe6a1d9b700 1 --2- 172.21.15.174:0/53575053 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe69002ffb0 0x7fe690032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:52.862 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.726+0000 7fe6a1d9b700 1 --2- 172.21.15.174:0/53575053 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe69002ffb0 0x7fe690032470 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fe68c0096a0 tx=0x7fe68c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:52.862 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.731+0000 7fe69affd700 1 -- 172.21.15.174:0/53575053 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe69406d2b0 con 0x7fe6a40f5410 2023-12-25T21:07:52.938 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.937+0000 7fe6a8da0700 1 -- 172.21.15.174:0/53575053 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6a40fae60 con 0x7fe6a40f5410 2023-12-25T21:07:52.938 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.938+0000 7fe69affd700 1 -- 172.21.15.174:0/53575053 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe6940095f0 con 0x7fe6a40f5410 2023-12-25T21:07:52.939 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:07:52.939 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:07:52.940 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.939+0000 7fe6a8da0700 1 -- 172.21.15.174:0/53575053 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe69002ffb0 msgr2=0x7fe690032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:52.940 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.939+0000 7fe6a8da0700 1 --2- 172.21.15.174:0/53575053 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe69002ffb0 0x7fe690032470 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fe68c0096a0 tx=0x7fe68c006b40).stop 2023-12-25T21:07:52.940 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.939+0000 7fe6a8da0700 1 -- 172.21.15.174:0/53575053 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a40f5410 msgr2=0x7fe6a4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:52.941 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.939+0000 7fe6a8da0700 1 --2- 172.21.15.174:0/53575053 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a40f5410 0x7fe6a4105630 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe694013510 tx=0x7fe69400b040).stop 2023-12-25T21:07:52.941 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.939+0000 7fe6a8da0700 1 -- 172.21.15.174:0/53575053 shutdown_connections 2023-12-25T21:07:52.941 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.939+0000 7fe6a8da0700 1 --2- 172.21.15.174:0/53575053 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe69002ffb0 0x7fe690032470 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:52.941 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.939+0000 7fe6a8da0700 1 --2- 172.21.15.174:0/53575053 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6a40f5410 0x7fe6a4105630 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:52.941 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.939+0000 7fe6a8da0700 1 -- 172.21.15.174:0/53575053 >> 172.21.15.174:0/53575053 conn(0x7fe6a40f0b70 msgr2=0x7fe6a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:52.941 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.940+0000 7fe6a8da0700 1 -- 172.21.15.174:0/53575053 shutdown_connections 2023-12-25T21:07:52.941 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:52.940+0000 7fe6a8da0700 1 -- 172.21.15.174:0/53575053 wait complete. 2023-12-25T21:07:52.941 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:07:53.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:52.701546+0000 mon.smithi061 (mon.0) 129 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/mirror_snapshot_schedule"}]: dispatch 2023-12-25T21:07:53.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:52.703756+0000 mon.smithi061 (mon.0) 130 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/trash_purge_schedule"}]: dispatch 2023-12-25T21:07:53.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:52.939187+0000 mon.smithi061 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.174:0/53575053' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:07:54.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:54 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:52.573803+0000 mgr.smithi061.jxxeke (mgr.14164) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:55.642 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:07:55.643 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:07:56.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:56 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:54.574273+0000 mgr.smithi061.jxxeke (mgr.14164) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:56.842 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.836+0000 7f6c9a3fb700 1 -- 172.21.15.174:0/3683942021 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c940f5410 msgr2=0x7f6c940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:56.842 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.836+0000 7f6c9a3fb700 1 --2- 172.21.15.174:0/3683942021 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c940f5410 0x7f6c940f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f6c88009fb0 tx=0x7f6c88009b30).stop 2023-12-25T21:07:56.842 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.836+0000 7f6c9a3fb700 1 -- 172.21.15.174:0/3683942021 shutdown_connections 2023-12-25T21:07:56.842 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.836+0000 7f6c9a3fb700 1 --2- 172.21.15.174:0/3683942021 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c940f5410 0x7f6c940f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:56.843 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.836+0000 7f6c9a3fb700 1 -- 172.21.15.174:0/3683942021 >> 172.21.15.174:0/3683942021 conn(0x7f6c940f0b70 msgr2=0x7f6c940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:56.843 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.837+0000 7f6c9a3fb700 1 -- 172.21.15.174:0/3683942021 shutdown_connections 2023-12-25T21:07:56.843 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.837+0000 7f6c9a3fb700 1 -- 172.21.15.174:0/3683942021 wait complete. 2023-12-25T21:07:56.843 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.837+0000 7f6c9a3fb700 1 Processor -- start 2023-12-25T21:07:56.844 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.837+0000 7f6c9a3fb700 1 -- start start 2023-12-25T21:07:56.844 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.837+0000 7f6c9a3fb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c940f5410 0x7f6c94109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:56.844 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.837+0000 7f6c9a3fb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c94060220 con 0x7f6c940f5410 2023-12-25T21:07:56.844 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.838+0000 7f6c93fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c940f5410 0x7f6c94109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:56.844 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.838+0000 7f6c93fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c940f5410 0x7f6c94109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:38556/0 (socket says 172.21.15.174:38556) 2023-12-25T21:07:56.844 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.838+0000 7f6c93fff700 1 -- 172.21.15.174:0/412091238 learned_addr learned my addr 172.21.15.174:0/412091238 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:07:56.845 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.838+0000 7f6c93fff700 1 -- 172.21.15.174:0/412091238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c88005040 con 0x7f6c940f5410 2023-12-25T21:07:56.845 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.838+0000 7f6c93fff700 1 --2- 172.21.15.174:0/412091238 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c940f5410 0x7f6c94109450 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f6c88013510 tx=0x7f6c8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:56.845 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.838+0000 7f6c90ff9700 1 -- 172.21.15.174:0/412091238 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c88009320 con 0x7f6c940f5410 2023-12-25T21:07:56.845 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.838+0000 7f6c90ff9700 1 -- 172.21.15.174:0/412091238 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c88013f00 con 0x7f6c940f5410 2023-12-25T21:07:56.845 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.838+0000 7f6c9a3fb700 1 -- 172.21.15.174:0/412091238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c94060420 con 0x7f6c940f5410 2023-12-25T21:07:56.845 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.838+0000 7f6c90ff9700 1 -- 172.21.15.174:0/412091238 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c88014d90 con 0x7f6c940f5410 2023-12-25T21:07:56.845 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.838+0000 7f6c9a3fb700 1 -- 172.21.15.174:0/412091238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c94109b00 con 0x7f6c940f5410 2023-12-25T21:07:56.845 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.839+0000 7f6c90ff9700 1 -- 172.21.15.174:0/412091238 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c880263a0 con 0x7f6c940f5410 2023-12-25T21:07:56.846 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.839+0000 7f6c90ff9700 1 --2- 172.21.15.174:0/412091238 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6c7c02ffb0 0x7f6c7c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:56.846 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.839+0000 7f6c9a3fb700 1 -- 172.21.15.174:0/412091238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c94059100 con 0x7f6c940f5410 2023-12-25T21:07:56.846 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.839+0000 7f6c90ff9700 1 -- 172.21.15.174:0/412091238 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c880410d0 con 0x7f6c940f5410 2023-12-25T21:07:56.846 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.839+0000 7f6c937fe700 1 --2- 172.21.15.174:0/412091238 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6c7c02ffb0 0x7f6c7c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:56.846 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.840+0000 7f6c937fe700 1 --2- 172.21.15.174:0/412091238 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6c7c02ffb0 0x7f6c7c032470 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f6c840096a0 tx=0x7f6c84006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:56.847 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:56.844+0000 7f6c90ff9700 1 -- 172.21.15.174:0/412091238 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c88012920 con 0x7f6c940f5410 2023-12-25T21:07:57.048 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:57.047+0000 7f6c9a3fb700 1 -- 172.21.15.174:0/412091238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c94045ad0 con 0x7f6c940f5410 2023-12-25T21:07:57.049 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:57.048+0000 7f6c90ff9700 1 -- 172.21.15.174:0/412091238 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6c8802b020 con 0x7f6c940f5410 2023-12-25T21:07:57.049 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:07:57.049 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:07:57.050 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:57.049+0000 7f6c9a3fb700 1 -- 172.21.15.174:0/412091238 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6c7c02ffb0 msgr2=0x7f6c7c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:57.050 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:57.049+0000 7f6c9a3fb700 1 --2- 172.21.15.174:0/412091238 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6c7c02ffb0 0x7f6c7c032470 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f6c840096a0 tx=0x7f6c84006b40).stop 2023-12-25T21:07:57.050 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:57.049+0000 7f6c9a3fb700 1 -- 172.21.15.174:0/412091238 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c940f5410 msgr2=0x7f6c94109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:57.050 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:57.049+0000 7f6c9a3fb700 1 --2- 172.21.15.174:0/412091238 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c940f5410 0x7f6c94109450 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f6c88013510 tx=0x7f6c8800b040).stop 2023-12-25T21:07:57.051 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:57.050+0000 7f6c9a3fb700 1 -- 172.21.15.174:0/412091238 shutdown_connections 2023-12-25T21:07:57.051 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:57.050+0000 7f6c9a3fb700 1 --2- 172.21.15.174:0/412091238 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6c7c02ffb0 0x7f6c7c032470 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:57.051 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:57.050+0000 7f6c9a3fb700 1 --2- 172.21.15.174:0/412091238 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c940f5410 0x7f6c94109450 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:57.051 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:57.050+0000 7f6c9a3fb700 1 -- 172.21.15.174:0/412091238 >> 172.21.15.174:0/412091238 conn(0x7f6c940f0b70 msgr2=0x7f6c940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:57.051 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:57.050+0000 7f6c9a3fb700 1 -- 172.21.15.174:0/412091238 shutdown_connections 2023-12-25T21:07:57.051 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:57.050+0000 7f6c9a3fb700 1 -- 172.21.15.174:0/412091238 wait complete. 2023-12-25T21:07:57.051 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:07:57.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:57 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:57.049365+0000 mon.smithi061 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.174:0/412091238' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:07:58.449 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:07:58.449 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:07:58.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:07:58 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:56.574646+0000 mgr.smithi061.jxxeke (mgr.14164) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:07:59.622 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.621+0000 7f4aa6046700 1 -- 172.21.15.174:0/3595860597 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4aa0060280 msgr2=0x7f4aa00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:59.623 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.621+0000 7f4aa6046700 1 --2- 172.21.15.174:0/3595860597 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4aa0060280 0x7f4aa00f90f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f4a88009fb0 tx=0x7f4a88009b30).stop 2023-12-25T21:07:59.623 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.621+0000 7f4aa6046700 1 -- 172.21.15.174:0/3595860597 shutdown_connections 2023-12-25T21:07:59.623 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.621+0000 7f4aa6046700 1 --2- 172.21.15.174:0/3595860597 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4aa0060280 0x7f4aa00f90f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:59.623 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.621+0000 7f4aa6046700 1 -- 172.21.15.174:0/3595860597 >> 172.21.15.174:0/3595860597 conn(0x7f4aa00f0b70 msgr2=0x7f4aa00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:59.623 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.621+0000 7f4aa6046700 1 -- 172.21.15.174:0/3595860597 shutdown_connections 2023-12-25T21:07:59.623 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.621+0000 7f4aa6046700 1 -- 172.21.15.174:0/3595860597 wait complete. 2023-12-25T21:07:59.623 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.622+0000 7f4aa6046700 1 Processor -- start 2023-12-25T21:07:59.623 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.622+0000 7f4aa6046700 1 -- start start 2023-12-25T21:07:59.623 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.622+0000 7f4aa6046700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4aa0060280 0x7f4aa01066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:59.623 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.622+0000 7f4aa6046700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa0106bb0 con 0x7f4aa0060280 2023-12-25T21:07:59.625 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.622+0000 7f4a9f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4aa0060280 0x7f4aa01066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:59.625 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.622+0000 7f4a9f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4aa0060280 0x7f4aa01066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:38572/0 (socket says 172.21.15.174:38572) 2023-12-25T21:07:59.625 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.622+0000 7f4a9f7fe700 1 -- 172.21.15.174:0/492132783 learned_addr learned my addr 172.21.15.174:0/492132783 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:07:59.625 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.623+0000 7f4a9f7fe700 1 -- 172.21.15.174:0/492132783 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a88005040 con 0x7f4aa0060280 2023-12-25T21:07:59.625 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.623+0000 7f4a9f7fe700 1 --2- 172.21.15.174:0/492132783 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4aa0060280 0x7f4aa01066c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f4a88013510 tx=0x7f4a8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:59.625 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.623+0000 7f4a97fff700 1 -- 172.21.15.174:0/492132783 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a88009320 con 0x7f4aa0060280 2023-12-25T21:07:59.625 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.623+0000 7f4aa6046700 1 -- 172.21.15.174:0/492132783 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa0106db0 con 0x7f4aa0060280 2023-12-25T21:07:59.626 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.623+0000 7f4a97fff700 1 -- 172.21.15.174:0/492132783 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a88013f00 con 0x7f4aa0060280 2023-12-25T21:07:59.626 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.623+0000 7f4a97fff700 1 -- 172.21.15.174:0/492132783 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a88014dc0 con 0x7f4aa0060280 2023-12-25T21:07:59.626 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.623+0000 7f4aa6046700 1 -- 172.21.15.174:0/492132783 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa0107240 con 0x7f4aa0060280 2023-12-25T21:07:59.626 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.624+0000 7f4a97fff700 1 -- 172.21.15.174:0/492132783 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a880095f0 con 0x7f4aa0060280 2023-12-25T21:07:59.626 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.624+0000 7f4a97fff700 1 --2- 172.21.15.174:0/492132783 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f4a8c02ff60 0x7f4a8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:07:59.626 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.624+0000 7f4aa6046700 1 -- 172.21.15.174:0/492132783 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa0045ad0 con 0x7f4aa0060280 2023-12-25T21:07:59.626 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.624+0000 7f4a97fff700 1 -- 172.21.15.174:0/492132783 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a88041000 con 0x7f4aa0060280 2023-12-25T21:07:59.626 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.624+0000 7f4a9effd700 1 --2- 172.21.15.174:0/492132783 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f4a8c02ff60 0x7f4a8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:07:59.626 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.625+0000 7f4a9effd700 1 --2- 172.21.15.174:0/492132783 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f4a8c02ff60 0x7f4a8c032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f4a900096a0 tx=0x7f4a90006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:07:59.629 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.629+0000 7f4a97fff700 1 -- 172.21.15.174:0/492132783 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a8806d2b0 con 0x7f4aa0060280 2023-12-25T21:07:59.837 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.836+0000 7f4aa6046700 1 -- 172.21.15.174:0/492132783 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4aa01080f0 con 0x7f4aa0060280 2023-12-25T21:07:59.838 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.837+0000 7f4a97fff700 1 -- 172.21.15.174:0/492132783 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4a88025350 con 0x7f4aa0060280 2023-12-25T21:07:59.838 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:07:59.838 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:07:59.839 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.839+0000 7f4aa6046700 1 -- 172.21.15.174:0/492132783 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f4a8c02ff60 msgr2=0x7f4a8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:59.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.839+0000 7f4aa6046700 1 --2- 172.21.15.174:0/492132783 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f4a8c02ff60 0x7f4a8c032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f4a900096a0 tx=0x7f4a90006b40).stop 2023-12-25T21:07:59.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.839+0000 7f4aa6046700 1 -- 172.21.15.174:0/492132783 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4aa0060280 msgr2=0x7f4aa01066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:07:59.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.839+0000 7f4aa6046700 1 --2- 172.21.15.174:0/492132783 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4aa0060280 0x7f4aa01066c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f4a88013510 tx=0x7f4a8800b040).stop 2023-12-25T21:07:59.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.839+0000 7f4aa6046700 1 -- 172.21.15.174:0/492132783 shutdown_connections 2023-12-25T21:07:59.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.839+0000 7f4aa6046700 1 --2- 172.21.15.174:0/492132783 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f4a8c02ff60 0x7f4a8c032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:59.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.839+0000 7f4aa6046700 1 --2- 172.21.15.174:0/492132783 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4aa0060280 0x7f4aa01066c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:07:59.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.839+0000 7f4aa6046700 1 -- 172.21.15.174:0/492132783 >> 172.21.15.174:0/492132783 conn(0x7f4aa00f0b70 msgr2=0x7f4aa00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:07:59.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.839+0000 7f4aa6046700 1 -- 172.21.15.174:0/492132783 shutdown_connections 2023-12-25T21:07:59.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:07:59.839+0000 7f4aa6046700 1 -- 172.21.15.174:0/492132783 wait complete. 2023-12-25T21:07:59.840 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:00.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:00 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:07:58.574961+0000 mgr.smithi061.jxxeke (mgr.14164) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:00.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:00 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:07:59.838512+0000 mon.smithi061 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.174:0/492132783' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:01.256 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:01.257 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:02.415 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.414+0000 7f092871a700 1 -- 172.21.15.174:0/1347975776 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09200f6510 msgr2=0x7f09200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:02.416 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.414+0000 7f092871a700 1 --2- 172.21.15.174:0/1347975776 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09200f6510 0x7f09200f6930 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f0910009fb0 tx=0x7f0910009b30).stop 2023-12-25T21:08:02.416 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.414+0000 7f092871a700 1 -- 172.21.15.174:0/1347975776 shutdown_connections 2023-12-25T21:08:02.416 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.414+0000 7f092871a700 1 --2- 172.21.15.174:0/1347975776 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09200f6510 0x7f09200f6930 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:02.416 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.414+0000 7f092871a700 1 -- 172.21.15.174:0/1347975776 >> 172.21.15.174:0/1347975776 conn(0x7f09200f1c70 msgr2=0x7f09200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:02.416 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.414+0000 7f092871a700 1 -- 172.21.15.174:0/1347975776 shutdown_connections 2023-12-25T21:08:02.416 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.414+0000 7f092871a700 1 -- 172.21.15.174:0/1347975776 wait complete. 2023-12-25T21:08:02.419 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.415+0000 7f092871a700 1 Processor -- start 2023-12-25T21:08:02.419 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.415+0000 7f092871a700 1 -- start start 2023-12-25T21:08:02.419 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.415+0000 7f092871a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09200f6510 0x7f0920106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:02.419 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.415+0000 7f092871a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0920106c20 con 0x7f09200f6510 2023-12-25T21:08:02.419 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.416+0000 7f09264b6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09200f6510 0x7f0920106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:02.419 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.416+0000 7f09264b6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09200f6510 0x7f0920106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:38588/0 (socket says 172.21.15.174:38588) 2023-12-25T21:08:02.420 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.416+0000 7f09264b6700 1 -- 172.21.15.174:0/456362962 learned_addr learned my addr 172.21.15.174:0/456362962 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:02.420 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.416+0000 7f09264b6700 1 -- 172.21.15.174:0/456362962 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0910005040 con 0x7f09200f6510 2023-12-25T21:08:02.420 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.416+0000 7f09264b6700 1 --2- 172.21.15.174:0/456362962 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09200f6510 0x7f0920106730 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f0910013510 tx=0x7f091000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:02.420 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.416+0000 7f0916ffd700 1 -- 172.21.15.174:0/456362962 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0910009320 con 0x7f09200f6510 2023-12-25T21:08:02.420 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.416+0000 7f0916ffd700 1 -- 172.21.15.174:0/456362962 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0910013f00 con 0x7f09200f6510 2023-12-25T21:08:02.420 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.416+0000 7f092871a700 1 -- 172.21.15.174:0/456362962 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0920106e20 con 0x7f09200f6510 2023-12-25T21:08:02.420 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.416+0000 7f0916ffd700 1 -- 172.21.15.174:0/456362962 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0910014d90 con 0x7f09200f6510 2023-12-25T21:08:02.421 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.416+0000 7f092871a700 1 -- 172.21.15.174:0/456362962 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09201072b0 con 0x7f09200f6510 2023-12-25T21:08:02.421 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.417+0000 7f0916ffd700 1 -- 172.21.15.174:0/456362962 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09100095f0 con 0x7f09200f6510 2023-12-25T21:08:02.421 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.417+0000 7f092871a700 1 -- 172.21.15.174:0/456362962 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0920045ad0 con 0x7f09200f6510 2023-12-25T21:08:02.421 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.417+0000 7f0916ffd700 1 --2- 172.21.15.174:0/456362962 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f090c02ff60 0x7f090c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:02.421 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.417+0000 7f0916ffd700 1 -- 172.21.15.174:0/456362962 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0910041000 con 0x7f09200f6510 2023-12-25T21:08:02.421 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.418+0000 7f0925cb5700 1 --2- 172.21.15.174:0/456362962 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f090c02ff60 0x7f090c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:02.421 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.418+0000 7f0925cb5700 1 --2- 172.21.15.174:0/456362962 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f090c02ff60 0x7f090c032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f091c0096a0 tx=0x7f091c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:02.423 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.422+0000 7f0916ffd700 1 -- 172.21.15.174:0/456362962 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0910014420 con 0x7f09200f6510 2023-12-25T21:08:02.628 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.628+0000 7f092871a700 1 -- 172.21.15.174:0/456362962 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09200fbf60 con 0x7f09200f6510 2023-12-25T21:08:02.629 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.628+0000 7f0916ffd700 1 -- 172.21.15.174:0/456362962 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f091001d7f0 con 0x7f09200f6510 2023-12-25T21:08:02.629 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:02.629 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:02.631 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.630+0000 7f092871a700 1 -- 172.21.15.174:0/456362962 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f090c02ff60 msgr2=0x7f090c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:02.631 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.630+0000 7f092871a700 1 --2- 172.21.15.174:0/456362962 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f090c02ff60 0x7f090c032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f091c0096a0 tx=0x7f091c006b40).stop 2023-12-25T21:08:02.631 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.630+0000 7f092871a700 1 -- 172.21.15.174:0/456362962 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09200f6510 msgr2=0x7f0920106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:02.631 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.630+0000 7f092871a700 1 --2- 172.21.15.174:0/456362962 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09200f6510 0x7f0920106730 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f0910013510 tx=0x7f091000b040).stop 2023-12-25T21:08:02.631 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.630+0000 7f092871a700 1 -- 172.21.15.174:0/456362962 shutdown_connections 2023-12-25T21:08:02.632 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.630+0000 7f092871a700 1 --2- 172.21.15.174:0/456362962 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f090c02ff60 0x7f090c032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:02.632 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.630+0000 7f092871a700 1 --2- 172.21.15.174:0/456362962 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09200f6510 0x7f0920106730 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:02.632 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.630+0000 7f092871a700 1 -- 172.21.15.174:0/456362962 >> 172.21.15.174:0/456362962 conn(0x7f09200f1c70 msgr2=0x7f09200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:02.632 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.630+0000 7f092871a700 1 -- 172.21.15.174:0/456362962 shutdown_connections 2023-12-25T21:08:02.632 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:02.630+0000 7f092871a700 1 -- 172.21.15.174:0/456362962 wait complete. 2023-12-25T21:08:02.632 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:02.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:02 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:00.575216+0000 mgr.smithi061.jxxeke (mgr.14164) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:03.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:02.629847+0000 mon.smithi061 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.174:0/456362962' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:04.007 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:04.008 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:04.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:04 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:02.575532+0000 mgr.smithi061.jxxeke (mgr.14164) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.121+0000 7f269d540700 1 -- 172.21.15.174:0/2327587804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26980f5140 msgr2=0x7f26980f5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.121+0000 7f269d540700 1 --2- 172.21.15.174:0/2327587804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26980f5140 0x7f26980f5560 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2688009fb0 tx=0x7f2688009b30).stop 2023-12-25T21:08:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.121+0000 7f269d540700 1 -- 172.21.15.174:0/2327587804 shutdown_connections 2023-12-25T21:08:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.121+0000 7f269d540700 1 --2- 172.21.15.174:0/2327587804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26980f5140 0x7f26980f5560 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.121+0000 7f269d540700 1 -- 172.21.15.174:0/2327587804 >> 172.21.15.174:0/2327587804 conn(0x7f26980f08c0 msgr2=0x7f26980f2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.121+0000 7f269d540700 1 -- 172.21.15.174:0/2327587804 shutdown_connections 2023-12-25T21:08:05.122 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.121+0000 7f269d540700 1 -- 172.21.15.174:0/2327587804 wait complete. 2023-12-25T21:08:05.123 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.121+0000 7f269d540700 1 Processor -- start 2023-12-25T21:08:05.123 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.121+0000 7f269d540700 1 -- start start 2023-12-25T21:08:05.123 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.121+0000 7f269d540700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26980f5140 0x7f26981064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:05.123 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.121+0000 7f269d540700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2698106990 con 0x7f26980f5140 2023-12-25T21:08:05.124 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.122+0000 7f2697fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26980f5140 0x7f26981064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:05.124 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.122+0000 7f2697fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26980f5140 0x7f26981064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:38612/0 (socket says 172.21.15.174:38612) 2023-12-25T21:08:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.122+0000 7f2697fff700 1 -- 172.21.15.174:0/376549 learned_addr learned my addr 172.21.15.174:0/376549 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.122+0000 7f2697fff700 1 -- 172.21.15.174:0/376549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2688005040 con 0x7f26980f5140 2023-12-25T21:08:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.122+0000 7f2697fff700 1 --2- 172.21.15.174:0/376549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26980f5140 0x7f26981064a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f2688014000 tx=0x7f268800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.122+0000 7f2694ff9700 1 -- 172.21.15.174:0/376549 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26880095f0 con 0x7f26980f5140 2023-12-25T21:08:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.122+0000 7f269d540700 1 -- 172.21.15.174:0/376549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2698106b90 con 0x7f26980f5140 2023-12-25T21:08:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.122+0000 7f2694ff9700 1 -- 172.21.15.174:0/376549 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2688007e70 con 0x7f26980f5140 2023-12-25T21:08:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.122+0000 7f2694ff9700 1 -- 172.21.15.174:0/376549 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2688014ea0 con 0x7f26980f5140 2023-12-25T21:08:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.122+0000 7f269d540700 1 -- 172.21.15.174:0/376549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2698107020 con 0x7f26980f5140 2023-12-25T21:08:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.123+0000 7f2694ff9700 1 -- 172.21.15.174:0/376549 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2688025070 con 0x7f26980f5140 2023-12-25T21:08:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.123+0000 7f2694ff9700 1 --2- 172.21.15.174:0/376549 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f268002ff60 0x7f2680032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.123+0000 7f2694ff9700 1 -- 172.21.15.174:0/376549 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2688040a50 con 0x7f26980f5140 2023-12-25T21:08:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.123+0000 7f269d540700 1 -- 172.21.15.174:0/376549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2698045ad0 con 0x7f26980f5140 2023-12-25T21:08:05.125 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.123+0000 7f26977fe700 1 --2- 172.21.15.174:0/376549 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f268002ff60 0x7f2680032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:05.126 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.124+0000 7f26977fe700 1 --2- 172.21.15.174:0/376549 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f268002ff60 0x7f2680032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f268c0096a0 tx=0x7f268c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:05.128 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.128+0000 7f2694ff9700 1 -- 172.21.15.174:0/376549 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2688026350 con 0x7f26980f5140 2023-12-25T21:08:05.335 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.334+0000 7f269d540700 1 -- 172.21.15.174:0/376549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2698107d40 con 0x7f26980f5140 2023-12-25T21:08:05.335 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.335+0000 7f2694ff9700 1 -- 172.21.15.174:0/376549 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2688025300 con 0x7f26980f5140 2023-12-25T21:08:05.336 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:05.336 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:05.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.336+0000 7f269d540700 1 -- 172.21.15.174:0/376549 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f268002ff60 msgr2=0x7f2680032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:05.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.336+0000 7f269d540700 1 --2- 172.21.15.174:0/376549 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f268002ff60 0x7f2680032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f268c0096a0 tx=0x7f268c006b40).stop 2023-12-25T21:08:05.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.336+0000 7f269d540700 1 -- 172.21.15.174:0/376549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26980f5140 msgr2=0x7f26981064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:05.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.336+0000 7f269d540700 1 --2- 172.21.15.174:0/376549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26980f5140 0x7f26981064a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f2688014000 tx=0x7f268800b040).stop 2023-12-25T21:08:05.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.336+0000 7f269d540700 1 -- 172.21.15.174:0/376549 shutdown_connections 2023-12-25T21:08:05.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.336+0000 7f269d540700 1 --2- 172.21.15.174:0/376549 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f268002ff60 0x7f2680032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:05.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.336+0000 7f269d540700 1 --2- 172.21.15.174:0/376549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26980f5140 0x7f26981064a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:05.338 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.336+0000 7f269d540700 1 -- 172.21.15.174:0/376549 >> 172.21.15.174:0/376549 conn(0x7f26980f08c0 msgr2=0x7f269805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:05.338 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.336+0000 7f269d540700 1 -- 172.21.15.174:0/376549 shutdown_connections 2023-12-25T21:08:05.338 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:05.336+0000 7f269d540700 1 -- 172.21.15.174:0/376549 wait complete. 2023-12-25T21:08:05.338 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:06.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:06 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:04.575959+0000 mgr.smithi061.jxxeke (mgr.14164) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:06.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:06 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:05.336188+0000 mon.smithi061 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.174:0/376549' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:06.720 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:06.720 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:07.903 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.902+0000 7f5e154ae700 1 -- 172.21.15.174:0/535029230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e100f5410 msgr2=0x7f5e100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:07.903 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.902+0000 7f5e154ae700 1 --2- 172.21.15.174:0/535029230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e100f5410 0x7f5e100f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f5df8009fb0 tx=0x7f5df8009b30).stop 2023-12-25T21:08:07.903 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.902+0000 7f5e154ae700 1 -- 172.21.15.174:0/535029230 shutdown_connections 2023-12-25T21:08:07.903 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.902+0000 7f5e154ae700 1 --2- 172.21.15.174:0/535029230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e100f5410 0x7f5e100f5830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:07.903 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.902+0000 7f5e154ae700 1 -- 172.21.15.174:0/535029230 >> 172.21.15.174:0/535029230 conn(0x7f5e100f0b70 msgr2=0x7f5e100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:07.903 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.902+0000 7f5e154ae700 1 -- 172.21.15.174:0/535029230 shutdown_connections 2023-12-25T21:08:07.903 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.902+0000 7f5e154ae700 1 -- 172.21.15.174:0/535029230 wait complete. 2023-12-25T21:08:07.905 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.903+0000 7f5e154ae700 1 Processor -- start 2023-12-25T21:08:07.905 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.903+0000 7f5e154ae700 1 -- start start 2023-12-25T21:08:07.905 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.903+0000 7f5e154ae700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e100f5410 0x7f5e10109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:07.905 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.903+0000 7f5e154ae700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10060240 con 0x7f5e100f5410 2023-12-25T21:08:07.905 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.903+0000 7f5e0effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e100f5410 0x7f5e10109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:07.905 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.903+0000 7f5e0effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e100f5410 0x7f5e10109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:51844/0 (socket says 172.21.15.174:51844) 2023-12-25T21:08:07.905 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.903+0000 7f5e0effd700 1 -- 172.21.15.174:0/2495069600 learned_addr learned my addr 172.21.15.174:0/2495069600 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:07.905 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.904+0000 7f5e0effd700 1 -- 172.21.15.174:0/2495069600 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df8005040 con 0x7f5e100f5410 2023-12-25T21:08:07.905 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.904+0000 7f5e0effd700 1 --2- 172.21.15.174:0/2495069600 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e100f5410 0x7f5e10109400 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f5df8013510 tx=0x7f5df800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:07.906 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.904+0000 7f5e077fe700 1 -- 172.21.15.174:0/2495069600 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5df8009320 con 0x7f5e100f5410 2023-12-25T21:08:07.906 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.904+0000 7f5e154ae700 1 -- 172.21.15.174:0/2495069600 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e10060440 con 0x7f5e100f5410 2023-12-25T21:08:07.906 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.904+0000 7f5e077fe700 1 -- 172.21.15.174:0/2495069600 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5df8013f00 con 0x7f5e100f5410 2023-12-25T21:08:07.906 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.904+0000 7f5e077fe700 1 -- 172.21.15.174:0/2495069600 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5df8014d90 con 0x7f5e100f5410 2023-12-25T21:08:07.906 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.904+0000 7f5e154ae700 1 -- 172.21.15.174:0/2495069600 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e10109ab0 con 0x7f5e100f5410 2023-12-25T21:08:07.907 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.905+0000 7f5e077fe700 1 -- 172.21.15.174:0/2495069600 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5df80095f0 con 0x7f5e100f5410 2023-12-25T21:08:07.907 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.905+0000 7f5e154ae700 1 -- 172.21.15.174:0/2495069600 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e10045ad0 con 0x7f5e100f5410 2023-12-25T21:08:07.907 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.905+0000 7f5e077fe700 1 --2- 172.21.15.174:0/2495069600 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5dfc02ff60 0x7f5dfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:07.907 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.905+0000 7f5e077fe700 1 -- 172.21.15.174:0/2495069600 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5df8041000 con 0x7f5e100f5410 2023-12-25T21:08:07.907 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.905+0000 7f5e0e7fc700 1 --2- 172.21.15.174:0/2495069600 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5dfc02ff60 0x7f5dfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:07.907 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.906+0000 7f5e0e7fc700 1 --2- 172.21.15.174:0/2495069600 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5dfc02ff60 0x7f5dfc032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f5e000096a0 tx=0x7f5e00006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:07.910 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:07.910+0000 7f5e077fe700 1 -- 172.21.15.174:0/2495069600 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5df8014420 con 0x7f5e100f5410 2023-12-25T21:08:08.119 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:08.117+0000 7f5e154ae700 1 -- 172.21.15.174:0/2495069600 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e100fae60 con 0x7f5e100f5410 2023-12-25T21:08:08.119 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:08.118+0000 7f5e077fe700 1 -- 172.21.15.174:0/2495069600 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5df8010230 con 0x7f5e100f5410 2023-12-25T21:08:08.119 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:08.119 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:08.121 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:08.119+0000 7f5e154ae700 1 -- 172.21.15.174:0/2495069600 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5dfc02ff60 msgr2=0x7f5dfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:08.121 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:08.119+0000 7f5e154ae700 1 --2- 172.21.15.174:0/2495069600 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5dfc02ff60 0x7f5dfc032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f5e000096a0 tx=0x7f5e00006b40).stop 2023-12-25T21:08:08.121 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:08.119+0000 7f5e154ae700 1 -- 172.21.15.174:0/2495069600 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e100f5410 msgr2=0x7f5e10109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:08.121 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:08.119+0000 7f5e154ae700 1 --2- 172.21.15.174:0/2495069600 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e100f5410 0x7f5e10109400 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f5df8013510 tx=0x7f5df800b040).stop 2023-12-25T21:08:08.122 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:08.119+0000 7f5e154ae700 1 -- 172.21.15.174:0/2495069600 shutdown_connections 2023-12-25T21:08:08.122 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:08.119+0000 7f5e154ae700 1 --2- 172.21.15.174:0/2495069600 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5dfc02ff60 0x7f5dfc032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:08.122 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:08.119+0000 7f5e154ae700 1 --2- 172.21.15.174:0/2495069600 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e100f5410 0x7f5e10109400 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:08.122 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:08.119+0000 7f5e154ae700 1 -- 172.21.15.174:0/2495069600 >> 172.21.15.174:0/2495069600 conn(0x7f5e100f0b70 msgr2=0x7f5e100f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:08.122 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:08.119+0000 7f5e154ae700 1 -- 172.21.15.174:0/2495069600 shutdown_connections 2023-12-25T21:08:08.123 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:08.119+0000 7f5e154ae700 1 -- 172.21.15.174:0/2495069600 wait complete. 2023-12-25T21:08:08.123 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:08.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:08 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:06.576456+0000 mgr.smithi061.jxxeke (mgr.14164) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:08.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:08 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:08.118985+0000 mon.smithi061 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.174:0/2495069600' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:09.475 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:09.475 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:10.656 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.655+0000 7fe118eaf700 1 -- 172.21.15.174:0/3864109223 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1140f62a0 msgr2=0x7fe1140f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:10.656 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.655+0000 7fe118eaf700 1 --2- 172.21.15.174:0/3864109223 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1140f62a0 0x7fe1140f66c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fe104009fb0 tx=0x7fe104009b30).stop 2023-12-25T21:08:10.657 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.655+0000 7fe118eaf700 1 -- 172.21.15.174:0/3864109223 shutdown_connections 2023-12-25T21:08:10.657 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.655+0000 7fe118eaf700 1 --2- 172.21.15.174:0/3864109223 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1140f62a0 0x7fe1140f66c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:10.657 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.655+0000 7fe118eaf700 1 -- 172.21.15.174:0/3864109223 >> 172.21.15.174:0/3864109223 conn(0x7fe1140f1aa0 msgr2=0x7fe1140f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:10.657 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.655+0000 7fe118eaf700 1 -- 172.21.15.174:0/3864109223 shutdown_connections 2023-12-25T21:08:10.657 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.655+0000 7fe118eaf700 1 -- 172.21.15.174:0/3864109223 wait complete. 2023-12-25T21:08:10.657 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.656+0000 7fe118eaf700 1 Processor -- start 2023-12-25T21:08:10.657 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.656+0000 7fe118eaf700 1 -- start start 2023-12-25T21:08:10.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.656+0000 7fe118eaf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1140f62a0 0x7fe1141064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:10.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.656+0000 7fe118eaf700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1141069a0 con 0x7fe1140f62a0 2023-12-25T21:08:10.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.656+0000 7fe1137fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1140f62a0 0x7fe1141064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:10.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.656+0000 7fe1137fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1140f62a0 0x7fe1141064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:51854/0 (socket says 172.21.15.174:51854) 2023-12-25T21:08:10.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.656+0000 7fe1137fe700 1 -- 172.21.15.174:0/866266757 learned_addr learned my addr 172.21.15.174:0/866266757 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:10.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.657+0000 7fe1137fe700 1 -- 172.21.15.174:0/866266757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe104005040 con 0x7fe1140f62a0 2023-12-25T21:08:10.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.657+0000 7fe1137fe700 1 --2- 172.21.15.174:0/866266757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1140f62a0 0x7fe1141064b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fe104013510 tx=0x7fe10400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:10.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.657+0000 7fe0fbfff700 1 -- 172.21.15.174:0/866266757 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe104009320 con 0x7fe1140f62a0 2023-12-25T21:08:10.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.657+0000 7fe118eaf700 1 -- 172.21.15.174:0/866266757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe114106ba0 con 0x7fe1140f62a0 2023-12-25T21:08:10.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.657+0000 7fe0fbfff700 1 -- 172.21.15.174:0/866266757 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe104013f00 con 0x7fe1140f62a0 2023-12-25T21:08:10.660 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.657+0000 7fe0fbfff700 1 -- 172.21.15.174:0/866266757 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe104014dc0 con 0x7fe1140f62a0 2023-12-25T21:08:10.660 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.657+0000 7fe118eaf700 1 -- 172.21.15.174:0/866266757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe114107030 con 0x7fe1140f62a0 2023-12-25T21:08:10.660 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.658+0000 7fe0fbfff700 1 -- 172.21.15.174:0/866266757 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1040095f0 con 0x7fe1140f62a0 2023-12-25T21:08:10.660 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.658+0000 7fe118eaf700 1 -- 172.21.15.174:0/866266757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe114045ad0 con 0x7fe1140f62a0 2023-12-25T21:08:10.660 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.658+0000 7fe0fbfff700 1 --2- 172.21.15.174:0/866266757 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe0fc02ff60 0x7fe0fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:10.660 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.658+0000 7fe0fbfff700 1 -- 172.21.15.174:0/866266757 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe104041000 con 0x7fe1140f62a0 2023-12-25T21:08:10.660 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.658+0000 7fe112ffd700 1 --2- 172.21.15.174:0/866266757 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe0fc02ff60 0x7fe0fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:10.660 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.659+0000 7fe112ffd700 1 --2- 172.21.15.174:0/866266757 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe0fc02ff60 0x7fe0fc032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fe1080096a0 tx=0x7fe108006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:10.663 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.663+0000 7fe0fbfff700 1 -- 172.21.15.174:0/866266757 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe104014420 con 0x7fe1140f62a0 2023-12-25T21:08:10.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:10 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:08.576954+0000 mgr.smithi061.jxxeke (mgr.14164) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:10.870 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.869+0000 7fe118eaf700 1 -- 172.21.15.174:0/866266757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe114107ee0 con 0x7fe1140f62a0 2023-12-25T21:08:10.871 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.870+0000 7fe0fbfff700 1 -- 172.21.15.174:0/866266757 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe104010230 con 0x7fe1140f62a0 2023-12-25T21:08:10.871 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:10.871 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:10.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.871+0000 7fe118eaf700 1 -- 172.21.15.174:0/866266757 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe0fc02ff60 msgr2=0x7fe0fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:10.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.871+0000 7fe118eaf700 1 --2- 172.21.15.174:0/866266757 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe0fc02ff60 0x7fe0fc032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fe1080096a0 tx=0x7fe108006b40).stop 2023-12-25T21:08:10.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.871+0000 7fe118eaf700 1 -- 172.21.15.174:0/866266757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1140f62a0 msgr2=0x7fe1141064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:10.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.871+0000 7fe118eaf700 1 --2- 172.21.15.174:0/866266757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1140f62a0 0x7fe1141064b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fe104013510 tx=0x7fe10400b040).stop 2023-12-25T21:08:10.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.871+0000 7fe118eaf700 1 -- 172.21.15.174:0/866266757 shutdown_connections 2023-12-25T21:08:10.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.871+0000 7fe118eaf700 1 --2- 172.21.15.174:0/866266757 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe0fc02ff60 0x7fe0fc032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:10.873 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.871+0000 7fe118eaf700 1 --2- 172.21.15.174:0/866266757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1140f62a0 0x7fe1141064b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:10.873 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.871+0000 7fe118eaf700 1 -- 172.21.15.174:0/866266757 >> 172.21.15.174:0/866266757 conn(0x7fe1140f1aa0 msgr2=0x7fe1140f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:10.873 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.871+0000 7fe118eaf700 1 -- 172.21.15.174:0/866266757 shutdown_connections 2023-12-25T21:08:10.873 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:10.871+0000 7fe118eaf700 1 -- 172.21.15.174:0/866266757 wait complete. 2023-12-25T21:08:10.873 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:11.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:11 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:10.871211+0000 mon.smithi061 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.174:0/866266757' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:12.273 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:12.273 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:12.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:12 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:10.577396+0000 mgr.smithi061.jxxeke (mgr.14164) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:13.462 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.459+0000 7ff80e76f700 1 -- 172.21.15.174:0/1313325399 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8080f5410 msgr2=0x7ff8080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:13.462 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.459+0000 7ff80e76f700 1 --2- 172.21.15.174:0/1313325399 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8080f5410 0x7ff8080f5830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7ff7f8009fb0 tx=0x7ff7f8009b30).stop 2023-12-25T21:08:13.462 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.459+0000 7ff80e76f700 1 -- 172.21.15.174:0/1313325399 shutdown_connections 2023-12-25T21:08:13.462 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.459+0000 7ff80e76f700 1 --2- 172.21.15.174:0/1313325399 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8080f5410 0x7ff8080f5830 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:13.462 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.459+0000 7ff80e76f700 1 -- 172.21.15.174:0/1313325399 >> 172.21.15.174:0/1313325399 conn(0x7ff8080f0b70 msgr2=0x7ff8080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:13.462 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.459+0000 7ff80e76f700 1 -- 172.21.15.174:0/1313325399 shutdown_connections 2023-12-25T21:08:13.463 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.459+0000 7ff80e76f700 1 -- 172.21.15.174:0/1313325399 wait complete. 2023-12-25T21:08:13.463 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.460+0000 7ff80e76f700 1 Processor -- start 2023-12-25T21:08:13.463 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.460+0000 7ff80e76f700 1 -- start start 2023-12-25T21:08:13.463 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.460+0000 7ff80e76f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8080f5410 0x7ff808105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:13.463 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.460+0000 7ff80e76f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff808105b20 con 0x7ff8080f5410 2023-12-25T21:08:13.463 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.460+0000 7ff807fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8080f5410 0x7ff808105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:13.463 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.460+0000 7ff807fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8080f5410 0x7ff808105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:51874/0 (socket says 172.21.15.174:51874) 2023-12-25T21:08:13.463 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.460+0000 7ff807fff700 1 -- 172.21.15.174:0/1604395619 learned_addr learned my addr 172.21.15.174:0/1604395619 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:13.463 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.461+0000 7ff807fff700 1 -- 172.21.15.174:0/1604395619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f8005040 con 0x7ff8080f5410 2023-12-25T21:08:13.463 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.461+0000 7ff807fff700 1 --2- 172.21.15.174:0/1604395619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8080f5410 0x7ff808105630 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7ff7f8013510 tx=0x7ff7f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:13.463 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.461+0000 7ff804ff9700 1 -- 172.21.15.174:0/1604395619 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7f8009320 con 0x7ff8080f5410 2023-12-25T21:08:13.464 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.461+0000 7ff80e76f700 1 -- 172.21.15.174:0/1604395619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff808105d20 con 0x7ff8080f5410 2023-12-25T21:08:13.464 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.461+0000 7ff804ff9700 1 -- 172.21.15.174:0/1604395619 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7f8013f00 con 0x7ff8080f5410 2023-12-25T21:08:13.464 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.461+0000 7ff804ff9700 1 -- 172.21.15.174:0/1604395619 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7f8014d90 con 0x7ff8080f5410 2023-12-25T21:08:13.464 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.461+0000 7ff80e76f700 1 -- 172.21.15.174:0/1604395619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8081061b0 con 0x7ff8080f5410 2023-12-25T21:08:13.465 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.462+0000 7ff804ff9700 1 -- 172.21.15.174:0/1604395619 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7f8014420 con 0x7ff8080f5410 2023-12-25T21:08:13.465 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.462+0000 7ff80e76f700 1 -- 172.21.15.174:0/1604395619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff808045ad0 con 0x7ff8080f5410 2023-12-25T21:08:13.465 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.462+0000 7ff804ff9700 1 --2- 172.21.15.174:0/1604395619 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff7f002ff60 0x7ff7f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:13.465 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.462+0000 7ff804ff9700 1 -- 172.21.15.174:0/1604395619 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff7f8040fb0 con 0x7ff8080f5410 2023-12-25T21:08:13.465 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.462+0000 7ff8077fe700 1 --2- 172.21.15.174:0/1604395619 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff7f002ff60 0x7ff7f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:13.465 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.463+0000 7ff8077fe700 1 --2- 172.21.15.174:0/1604395619 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff7f002ff60 0x7ff7f0032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff7fc0096a0 tx=0x7ff7fc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:13.468 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.467+0000 7ff804ff9700 1 -- 172.21.15.174:0/1604395619 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff7f80095f0 con 0x7ff8080f5410 2023-12-25T21:08:13.674 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.673+0000 7ff80e76f700 1 -- 172.21.15.174:0/1604395619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8080fae60 con 0x7ff8080f5410 2023-12-25T21:08:13.674 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.674+0000 7ff804ff9700 1 -- 172.21.15.174:0/1604395619 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff7f801db70 con 0x7ff8080f5410 2023-12-25T21:08:13.675 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:13.675 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:13.676 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.675+0000 7ff80e76f700 1 -- 172.21.15.174:0/1604395619 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff7f002ff60 msgr2=0x7ff7f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:13.676 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.675+0000 7ff80e76f700 1 --2- 172.21.15.174:0/1604395619 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff7f002ff60 0x7ff7f0032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff7fc0096a0 tx=0x7ff7fc006b40).stop 2023-12-25T21:08:13.676 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.675+0000 7ff80e76f700 1 -- 172.21.15.174:0/1604395619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8080f5410 msgr2=0x7ff808105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:13.676 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.675+0000 7ff80e76f700 1 --2- 172.21.15.174:0/1604395619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8080f5410 0x7ff808105630 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7ff7f8013510 tx=0x7ff7f800b040).stop 2023-12-25T21:08:13.676 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.675+0000 7ff80e76f700 1 -- 172.21.15.174:0/1604395619 shutdown_connections 2023-12-25T21:08:13.676 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.675+0000 7ff80e76f700 1 --2- 172.21.15.174:0/1604395619 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff7f002ff60 0x7ff7f0032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:13.676 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.675+0000 7ff80e76f700 1 --2- 172.21.15.174:0/1604395619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8080f5410 0x7ff808105630 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:13.677 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.675+0000 7ff80e76f700 1 -- 172.21.15.174:0/1604395619 >> 172.21.15.174:0/1604395619 conn(0x7ff8080f0b70 msgr2=0x7ff8080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:13.677 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.675+0000 7ff80e76f700 1 -- 172.21.15.174:0/1604395619 shutdown_connections 2023-12-25T21:08:13.677 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:13.675+0000 7ff80e76f700 1 -- 172.21.15.174:0/1604395619 wait complete. 2023-12-25T21:08:13.677 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:14.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:14 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:12.577696+0000 mgr.smithi061.jxxeke (mgr.14164) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:14.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:14 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:13.675079+0000 mon.smithi061 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.174:0/1604395619' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:15.084 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:15.085 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:16.296 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.294+0000 7fba6892a700 1 -- 172.21.15.174:0/2304993042 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba640f6510 msgr2=0x7fba640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:16.296 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.294+0000 7fba6892a700 1 --2- 172.21.15.174:0/2304993042 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba640f6510 0x7fba640f6930 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fba54009960 tx=0x7fba54004d10).stop 2023-12-25T21:08:16.296 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.294+0000 7fba6892a700 1 -- 172.21.15.174:0/2304993042 shutdown_connections 2023-12-25T21:08:16.296 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.294+0000 7fba6892a700 1 --2- 172.21.15.174:0/2304993042 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba640f6510 0x7fba640f6930 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:16.296 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.294+0000 7fba6892a700 1 -- 172.21.15.174:0/2304993042 >> 172.21.15.174:0/2304993042 conn(0x7fba640f1c70 msgr2=0x7fba640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:16.296 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.294+0000 7fba6892a700 1 -- 172.21.15.174:0/2304993042 shutdown_connections 2023-12-25T21:08:16.296 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.294+0000 7fba6892a700 1 -- 172.21.15.174:0/2304993042 wait complete. 2023-12-25T21:08:16.296 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.294+0000 7fba6892a700 1 Processor -- start 2023-12-25T21:08:16.296 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.294+0000 7fba6892a700 1 -- start start 2023-12-25T21:08:16.297 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.294+0000 7fba6892a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba640f6510 0x7fba64106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:16.297 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.294+0000 7fba6892a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba64106c20 con 0x7fba640f6510 2023-12-25T21:08:16.297 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.295+0000 7fba6259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba640f6510 0x7fba64106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:16.297 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.295+0000 7fba6259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba640f6510 0x7fba64106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:51882/0 (socket says 172.21.15.174:51882) 2023-12-25T21:08:16.297 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.295+0000 7fba6259c700 1 -- 172.21.15.174:0/1292069056 learned_addr learned my addr 172.21.15.174:0/1292069056 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:16.297 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.295+0000 7fba6259c700 1 -- 172.21.15.174:0/1292069056 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba54009640 con 0x7fba640f6510 2023-12-25T21:08:16.297 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.295+0000 7fba6259c700 1 --2- 172.21.15.174:0/1292069056 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba640f6510 0x7fba64106730 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fba54004b70 tx=0x7fba54004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:16.297 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.295+0000 7fba5affd700 1 -- 172.21.15.174:0/1292069056 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba54018070 con 0x7fba640f6510 2023-12-25T21:08:16.297 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.295+0000 7fba6892a700 1 -- 172.21.15.174:0/1292069056 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba64106e20 con 0x7fba640f6510 2023-12-25T21:08:16.297 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.295+0000 7fba5affd700 1 -- 172.21.15.174:0/1292069056 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba5400b870 con 0x7fba640f6510 2023-12-25T21:08:16.297 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.295+0000 7fba5affd700 1 -- 172.21.15.174:0/1292069056 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba5401c560 con 0x7fba640f6510 2023-12-25T21:08:16.297 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.295+0000 7fba6892a700 1 -- 172.21.15.174:0/1292069056 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba641072b0 con 0x7fba640f6510 2023-12-25T21:08:16.298 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.296+0000 7fba5affd700 1 -- 172.21.15.174:0/1292069056 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba54013630 con 0x7fba640f6510 2023-12-25T21:08:16.298 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.296+0000 7fba6892a700 1 -- 172.21.15.174:0/1292069056 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba64045ad0 con 0x7fba640f6510 2023-12-25T21:08:16.298 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.296+0000 7fba5affd700 1 --2- 172.21.15.174:0/1292069056 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fba5002ff60 0x7fba50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:16.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.296+0000 7fba5affd700 1 -- 172.21.15.174:0/1292069056 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba54041280 con 0x7fba640f6510 2023-12-25T21:08:16.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.296+0000 7fba61d9b700 1 --2- 172.21.15.174:0/1292069056 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fba5002ff60 0x7fba50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:16.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.297+0000 7fba61d9b700 1 --2- 172.21.15.174:0/1292069056 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fba5002ff60 0x7fba50032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fba4c0096a0 tx=0x7fba4c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:16.301 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.301+0000 7fba5affd700 1 -- 172.21.15.174:0/1292069056 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba54004810 con 0x7fba640f6510 2023-12-25T21:08:16.508 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.507+0000 7fba6892a700 1 -- 172.21.15.174:0/1292069056 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba640fbf60 con 0x7fba640f6510 2023-12-25T21:08:16.508 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.508+0000 7fba5affd700 1 -- 172.21.15.174:0/1292069056 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fba54012380 con 0x7fba640f6510 2023-12-25T21:08:16.509 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:16.509 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:16.510 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.509+0000 7fba6892a700 1 -- 172.21.15.174:0/1292069056 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fba5002ff60 msgr2=0x7fba50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:16.510 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.509+0000 7fba6892a700 1 --2- 172.21.15.174:0/1292069056 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fba5002ff60 0x7fba50032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fba4c0096a0 tx=0x7fba4c006b40).stop 2023-12-25T21:08:16.510 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.509+0000 7fba6892a700 1 -- 172.21.15.174:0/1292069056 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba640f6510 msgr2=0x7fba64106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:16.511 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.509+0000 7fba6892a700 1 --2- 172.21.15.174:0/1292069056 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba640f6510 0x7fba64106730 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fba54004b70 tx=0x7fba54004d10).stop 2023-12-25T21:08:16.511 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.510+0000 7fba6892a700 1 -- 172.21.15.174:0/1292069056 shutdown_connections 2023-12-25T21:08:16.511 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.510+0000 7fba6892a700 1 --2- 172.21.15.174:0/1292069056 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fba5002ff60 0x7fba50032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:16.511 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.510+0000 7fba6892a700 1 --2- 172.21.15.174:0/1292069056 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba640f6510 0x7fba64106730 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:16.511 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.510+0000 7fba6892a700 1 -- 172.21.15.174:0/1292069056 >> 172.21.15.174:0/1292069056 conn(0x7fba640f1c70 msgr2=0x7fba640f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:16.511 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.510+0000 7fba6892a700 1 -- 172.21.15.174:0/1292069056 shutdown_connections 2023-12-25T21:08:16.511 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:16.510+0000 7fba6892a700 1 -- 172.21.15.174:0/1292069056 wait complete. 2023-12-25T21:08:16.511 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:16.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:16 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:14.578190+0000 mgr.smithi061.jxxeke (mgr.14164) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:17.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:17 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:16.509075+0000 mon.smithi061 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.174:0/1292069056' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:17.919 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:17.920 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:18.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:18 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:16.578501+0000 mgr.smithi061.jxxeke (mgr.14164) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:19.109 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.108+0000 7f95b3fff700 1 -- 172.21.15.174:0/2927348349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95b40f51a0 msgr2=0x7f95b40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:19.109 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.108+0000 7f95b3fff700 1 --2- 172.21.15.174:0/2927348349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95b40f51a0 0x7f95b40f55c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f95a4009fb0 tx=0x7f95a4009b30).stop 2023-12-25T21:08:19.109 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.108+0000 7f95b3fff700 1 -- 172.21.15.174:0/2927348349 shutdown_connections 2023-12-25T21:08:19.109 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.108+0000 7f95b3fff700 1 --2- 172.21.15.174:0/2927348349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95b40f51a0 0x7f95b40f55c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:19.109 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.108+0000 7f95b3fff700 1 -- 172.21.15.174:0/2927348349 >> 172.21.15.174:0/2927348349 conn(0x7f95b40f09a0 msgr2=0x7f95b40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:19.109 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.108+0000 7f95b3fff700 1 -- 172.21.15.174:0/2927348349 shutdown_connections 2023-12-25T21:08:19.109 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.108+0000 7f95b3fff700 1 -- 172.21.15.174:0/2927348349 wait complete. 2023-12-25T21:08:19.111 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.109+0000 7f95b3fff700 1 Processor -- start 2023-12-25T21:08:19.111 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.109+0000 7f95b3fff700 1 -- start start 2023-12-25T21:08:19.111 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.109+0000 7f95b3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95b40f51a0 0x7f95b41091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:19.111 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.109+0000 7f95b3fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b4060220 con 0x7f95b40f51a0 2023-12-25T21:08:19.111 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.109+0000 7f95b2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95b40f51a0 0x7f95b41091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:19.111 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.109+0000 7f95b2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95b40f51a0 0x7f95b41091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:43022/0 (socket says 172.21.15.174:43022) 2023-12-25T21:08:19.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.109+0000 7f95b2ffd700 1 -- 172.21.15.174:0/3247279240 learned_addr learned my addr 172.21.15.174:0/3247279240 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:19.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.110+0000 7f95b2ffd700 1 -- 172.21.15.174:0/3247279240 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a4005040 con 0x7f95b40f51a0 2023-12-25T21:08:19.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.110+0000 7f95b2ffd700 1 --2- 172.21.15.174:0/3247279240 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95b40f51a0 0x7f95b41091d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f95a4013510 tx=0x7f95a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:19.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.110+0000 7f959b7fe700 1 -- 172.21.15.174:0/3247279240 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95a4009320 con 0x7f95b40f51a0 2023-12-25T21:08:19.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.110+0000 7f95b3fff700 1 -- 172.21.15.174:0/3247279240 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95b4060420 con 0x7f95b40f51a0 2023-12-25T21:08:19.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.110+0000 7f959b7fe700 1 -- 172.21.15.174:0/3247279240 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f95a4013f00 con 0x7f95b40f51a0 2023-12-25T21:08:19.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.110+0000 7f959b7fe700 1 -- 172.21.15.174:0/3247279240 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95a4014dc0 con 0x7f95b40f51a0 2023-12-25T21:08:19.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.110+0000 7f95b3fff700 1 -- 172.21.15.174:0/3247279240 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95b4109880 con 0x7f95b40f51a0 2023-12-25T21:08:19.113 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.111+0000 7f959b7fe700 1 -- 172.21.15.174:0/3247279240 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f95a40095f0 con 0x7f95b40f51a0 2023-12-25T21:08:19.113 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.111+0000 7f95b3fff700 1 -- 172.21.15.174:0/3247279240 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95b4059100 con 0x7f95b40f51a0 2023-12-25T21:08:19.113 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.111+0000 7f959b7fe700 1 --2- 172.21.15.174:0/3247279240 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f959c02ff60 0x7f959c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:19.113 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.111+0000 7f959b7fe700 1 -- 172.21.15.174:0/3247279240 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f95a4041000 con 0x7f95b40f51a0 2023-12-25T21:08:19.113 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.112+0000 7f95b27fc700 1 --2- 172.21.15.174:0/3247279240 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f959c02ff60 0x7f959c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:19.113 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.112+0000 7f95b27fc700 1 --2- 172.21.15.174:0/3247279240 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f959c02ff60 0x7f959c032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f95a80096a0 tx=0x7f95a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:19.117 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.116+0000 7f959b7fe700 1 -- 172.21.15.174:0/3247279240 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f95a401c790 con 0x7f95b40f51a0 2023-12-25T21:08:19.322 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.321+0000 7f95b3fff700 1 -- 172.21.15.174:0/3247279240 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95b4045ad0 con 0x7f95b40f51a0 2023-12-25T21:08:19.323 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.322+0000 7f959b7fe700 1 -- 172.21.15.174:0/3247279240 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f95a4010230 con 0x7f95b40f51a0 2023-12-25T21:08:19.323 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:19.323 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:19.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.323+0000 7f95b3fff700 1 -- 172.21.15.174:0/3247279240 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f959c02ff60 msgr2=0x7f959c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:19.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.323+0000 7f95b3fff700 1 --2- 172.21.15.174:0/3247279240 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f959c02ff60 0x7f959c032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f95a80096a0 tx=0x7f95a8006b40).stop 2023-12-25T21:08:19.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.323+0000 7f95b3fff700 1 -- 172.21.15.174:0/3247279240 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95b40f51a0 msgr2=0x7f95b41091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:19.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.323+0000 7f95b3fff700 1 --2- 172.21.15.174:0/3247279240 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95b40f51a0 0x7f95b41091d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f95a4013510 tx=0x7f95a400b040).stop 2023-12-25T21:08:19.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.323+0000 7f95b3fff700 1 -- 172.21.15.174:0/3247279240 shutdown_connections 2023-12-25T21:08:19.325 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.323+0000 7f95b3fff700 1 --2- 172.21.15.174:0/3247279240 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f959c02ff60 0x7f959c032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:19.325 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.323+0000 7f95b3fff700 1 --2- 172.21.15.174:0/3247279240 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95b40f51a0 0x7f95b41091d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:19.325 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.323+0000 7f95b3fff700 1 -- 172.21.15.174:0/3247279240 >> 172.21.15.174:0/3247279240 conn(0x7f95b40f09a0 msgr2=0x7f95b40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:19.325 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.323+0000 7f95b3fff700 1 -- 172.21.15.174:0/3247279240 shutdown_connections 2023-12-25T21:08:19.325 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:19.324+0000 7f95b3fff700 1 -- 172.21.15.174:0/3247279240 wait complete. 2023-12-25T21:08:19.325 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:20.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:20 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:18.578915+0000 mgr.smithi061.jxxeke (mgr.14164) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:20.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:20 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:19.323298+0000 mon.smithi061 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.174:0/3247279240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:20.740 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:20.740 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:21.902 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.901+0000 7f5b5de7b700 1 -- 172.21.15.174:0/635207334 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b58060220 msgr2=0x7f5b580f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:21.903 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.901+0000 7f5b5de7b700 1 --2- 172.21.15.174:0/635207334 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b58060220 0x7f5b580f9150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f5b48009fb0 tx=0x7f5b48009b30).stop 2023-12-25T21:08:21.903 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.901+0000 7f5b5de7b700 1 -- 172.21.15.174:0/635207334 shutdown_connections 2023-12-25T21:08:21.903 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.901+0000 7f5b5de7b700 1 --2- 172.21.15.174:0/635207334 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b58060220 0x7f5b580f9150 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:21.903 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.901+0000 7f5b5de7b700 1 -- 172.21.15.174:0/635207334 >> 172.21.15.174:0/635207334 conn(0x7f5b580f0b70 msgr2=0x7f5b580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:21.903 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.901+0000 7f5b5de7b700 1 -- 172.21.15.174:0/635207334 shutdown_connections 2023-12-25T21:08:21.903 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.901+0000 7f5b5de7b700 1 -- 172.21.15.174:0/635207334 wait complete. 2023-12-25T21:08:21.903 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.902+0000 7f5b5de7b700 1 Processor -- start 2023-12-25T21:08:21.903 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.902+0000 7f5b5de7b700 1 -- start start 2023-12-25T21:08:21.903 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.902+0000 7f5b5de7b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b58060220 0x7f5b580f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:21.903 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.902+0000 7f5b5de7b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b580f5050 con 0x7f5b58060220 2023-12-25T21:08:21.905 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.902+0000 7f5b577fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b58060220 0x7f5b580f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:21.905 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.902+0000 7f5b577fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b58060220 0x7f5b580f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:43038/0 (socket says 172.21.15.174:43038) 2023-12-25T21:08:21.905 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.902+0000 7f5b577fe700 1 -- 172.21.15.174:0/3199088086 learned_addr learned my addr 172.21.15.174:0/3199088086 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:21.905 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.903+0000 7f5b577fe700 1 -- 172.21.15.174:0/3199088086 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b48005040 con 0x7f5b58060220 2023-12-25T21:08:21.905 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.903+0000 7f5b577fe700 1 --2- 172.21.15.174:0/3199088086 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b58060220 0x7f5b580f6bd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f5b48013510 tx=0x7f5b4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:21.905 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.903+0000 7f5b4ffff700 1 -- 172.21.15.174:0/3199088086 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b48009320 con 0x7f5b58060220 2023-12-25T21:08:21.905 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.903+0000 7f5b5de7b700 1 -- 172.21.15.174:0/3199088086 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b580f51f0 con 0x7f5b58060220 2023-12-25T21:08:21.905 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.903+0000 7f5b4ffff700 1 -- 172.21.15.174:0/3199088086 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b48013f00 con 0x7f5b58060220 2023-12-25T21:08:21.905 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.903+0000 7f5b4ffff700 1 -- 172.21.15.174:0/3199088086 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b48014dc0 con 0x7f5b58060220 2023-12-25T21:08:21.905 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.903+0000 7f5b5de7b700 1 -- 172.21.15.174:0/3199088086 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b580f5680 con 0x7f5b58060220 2023-12-25T21:08:21.906 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.904+0000 7f5b4ffff700 1 -- 172.21.15.174:0/3199088086 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5b480095f0 con 0x7f5b58060220 2023-12-25T21:08:21.906 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.904+0000 7f5b5de7b700 1 -- 172.21.15.174:0/3199088086 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b58059100 con 0x7f5b58060220 2023-12-25T21:08:21.906 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.904+0000 7f5b4ffff700 1 --2- 172.21.15.174:0/3199088086 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5b4402ff60 0x7f5b44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:21.906 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.904+0000 7f5b4ffff700 1 -- 172.21.15.174:0/3199088086 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5b480410c0 con 0x7f5b58060220 2023-12-25T21:08:21.906 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.904+0000 7f5b56ffd700 1 --2- 172.21.15.174:0/3199088086 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5b4402ff60 0x7f5b44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:21.906 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.905+0000 7f5b56ffd700 1 --2- 172.21.15.174:0/3199088086 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5b4402ff60 0x7f5b44032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f5b400096a0 tx=0x7f5b40006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:21.909 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:21.909+0000 7f5b4ffff700 1 -- 172.21.15.174:0/3199088086 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b4801c790 con 0x7f5b58060220 2023-12-25T21:08:22.114 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:22.113+0000 7f5b5de7b700 1 -- 172.21.15.174:0/3199088086 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b580fbf50 con 0x7f5b58060220 2023-12-25T21:08:22.115 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:22.114+0000 7f5b4ffff700 1 -- 172.21.15.174:0/3199088086 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5b48010230 con 0x7f5b58060220 2023-12-25T21:08:22.115 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:22.115 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:22.116 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:22.116+0000 7f5b5de7b700 1 -- 172.21.15.174:0/3199088086 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5b4402ff60 msgr2=0x7f5b44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:22.117 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:22.116+0000 7f5b5de7b700 1 --2- 172.21.15.174:0/3199088086 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5b4402ff60 0x7f5b44032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f5b400096a0 tx=0x7f5b40006b40).stop 2023-12-25T21:08:22.117 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:22.116+0000 7f5b5de7b700 1 -- 172.21.15.174:0/3199088086 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b58060220 msgr2=0x7f5b580f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:22.117 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:22.116+0000 7f5b5de7b700 1 --2- 172.21.15.174:0/3199088086 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b58060220 0x7f5b580f6bd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f5b48013510 tx=0x7f5b4800b040).stop 2023-12-25T21:08:22.117 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:22.116+0000 7f5b5de7b700 1 -- 172.21.15.174:0/3199088086 shutdown_connections 2023-12-25T21:08:22.117 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:22.116+0000 7f5b5de7b700 1 --2- 172.21.15.174:0/3199088086 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5b4402ff60 0x7f5b44032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:22.117 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:22.116+0000 7f5b5de7b700 1 --2- 172.21.15.174:0/3199088086 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b58060220 0x7f5b580f6bd0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:22.117 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:22.116+0000 7f5b5de7b700 1 -- 172.21.15.174:0/3199088086 >> 172.21.15.174:0/3199088086 conn(0x7f5b580f0b70 msgr2=0x7f5b580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:22.117 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:22.116+0000 7f5b5de7b700 1 -- 172.21.15.174:0/3199088086 shutdown_connections 2023-12-25T21:08:22.117 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:22.116+0000 7f5b5de7b700 1 -- 172.21.15.174:0/3199088086 wait complete. 2023-12-25T21:08:22.118 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:22.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:22 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:20.579301+0000 mgr.smithi061.jxxeke (mgr.14164) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:22.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:22 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:22.115417+0000 mon.smithi061 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.174:0/3199088086' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:23.541 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:23.541 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:24.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:24 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:22.579585+0000 mgr.smithi061.jxxeke (mgr.14164) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:24.710 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.708+0000 7fa16f078700 1 -- 172.21.15.174:0/43154040 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa168060200 msgr2=0x7fa1680fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:24.710 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.708+0000 7fa16f078700 1 --2- 172.21.15.174:0/43154040 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa168060200 0x7fa1680fb570 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fa15c009fb0 tx=0x7fa15c009b30).stop 2023-12-25T21:08:24.710 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.708+0000 7fa16f078700 1 -- 172.21.15.174:0/43154040 shutdown_connections 2023-12-25T21:08:24.710 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.708+0000 7fa16f078700 1 --2- 172.21.15.174:0/43154040 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa168060200 0x7fa1680fb570 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:24.710 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.708+0000 7fa16f078700 1 -- 172.21.15.174:0/43154040 >> 172.21.15.174:0/43154040 conn(0x7fa1680f0b70 msgr2=0x7fa1680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:24.710 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.708+0000 7fa16f078700 1 -- 172.21.15.174:0/43154040 shutdown_connections 2023-12-25T21:08:24.710 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.708+0000 7fa16f078700 1 -- 172.21.15.174:0/43154040 wait complete. 2023-12-25T21:08:24.711 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.709+0000 7fa16f078700 1 Processor -- start 2023-12-25T21:08:24.711 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.709+0000 7fa16f078700 1 -- start start 2023-12-25T21:08:24.713 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.710+0000 7fa16f078700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa168060200 0x7fa1681088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:24.713 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.710+0000 7fa16f078700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa168108dc0 con 0x7fa168060200 2023-12-25T21:08:24.713 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.710+0000 7fa16ce14700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa168060200 0x7fa1681088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:24.714 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.710+0000 7fa16ce14700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa168060200 0x7fa1681088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:43046/0 (socket says 172.21.15.174:43046) 2023-12-25T21:08:24.714 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.710+0000 7fa16ce14700 1 -- 172.21.15.174:0/1545635995 learned_addr learned my addr 172.21.15.174:0/1545635995 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:24.714 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.711+0000 7fa16ce14700 1 -- 172.21.15.174:0/1545635995 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15c005040 con 0x7fa168060200 2023-12-25T21:08:24.714 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.711+0000 7fa16ce14700 1 --2- 172.21.15.174:0/1545635995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa168060200 0x7fa1681088d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fa15c013510 tx=0x7fa15c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:24.714 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.711+0000 7fa1657fa700 1 -- 172.21.15.174:0/1545635995 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa15c009320 con 0x7fa168060200 2023-12-25T21:08:24.714 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.711+0000 7fa1657fa700 1 -- 172.21.15.174:0/1545635995 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa15c013f00 con 0x7fa168060200 2023-12-25T21:08:24.714 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.711+0000 7fa16f078700 1 -- 172.21.15.174:0/1545635995 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa168108fc0 con 0x7fa168060200 2023-12-25T21:08:24.714 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.711+0000 7fa1657fa700 1 -- 172.21.15.174:0/1545635995 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa15c014dc0 con 0x7fa168060200 2023-12-25T21:08:24.714 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.711+0000 7fa16f078700 1 -- 172.21.15.174:0/1545635995 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa168109450 con 0x7fa168060200 2023-12-25T21:08:24.714 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.712+0000 7fa1657fa700 1 -- 172.21.15.174:0/1545635995 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa15c0263a0 con 0x7fa168060200 2023-12-25T21:08:24.714 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.712+0000 7fa16f078700 1 -- 172.21.15.174:0/1545635995 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa168045ad0 con 0x7fa168060200 2023-12-25T21:08:24.714 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.712+0000 7fa1657fa700 1 --2- 172.21.15.174:0/1545635995 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa15802ffb0 0x7fa158032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:24.715 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.712+0000 7fa1657fa700 1 -- 172.21.15.174:0/1545635995 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa15c0412c0 con 0x7fa168060200 2023-12-25T21:08:24.715 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.712+0000 7fa167fff700 1 --2- 172.21.15.174:0/1545635995 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa15802ffb0 0x7fa158032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:24.715 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.712+0000 7fa167fff700 1 --2- 172.21.15.174:0/1545635995 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa15802ffb0 0x7fa158032470 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fa1540096a0 tx=0x7fa154006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:24.717 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.716+0000 7fa1657fa700 1 -- 172.21.15.174:0/1545635995 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa15c01cb10 con 0x7fa168060200 2023-12-25T21:08:24.925 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.924+0000 7fa16f078700 1 -- 172.21.15.174:0/1545635995 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa16810a300 con 0x7fa168060200 2023-12-25T21:08:24.925 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.924+0000 7fa1657fa700 1 -- 172.21.15.174:0/1545635995 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa15c01cb10 con 0x7fa168060200 2023-12-25T21:08:24.926 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:24.926 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:24.928 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.926+0000 7fa16f078700 1 -- 172.21.15.174:0/1545635995 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa15802ffb0 msgr2=0x7fa158032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:24.928 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.926+0000 7fa16f078700 1 --2- 172.21.15.174:0/1545635995 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa15802ffb0 0x7fa158032470 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fa1540096a0 tx=0x7fa154006b40).stop 2023-12-25T21:08:24.928 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.926+0000 7fa16f078700 1 -- 172.21.15.174:0/1545635995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa168060200 msgr2=0x7fa1681088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:24.928 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.926+0000 7fa16f078700 1 --2- 172.21.15.174:0/1545635995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa168060200 0x7fa1681088d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fa15c013510 tx=0x7fa15c00b040).stop 2023-12-25T21:08:24.929 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.926+0000 7fa16f078700 1 -- 172.21.15.174:0/1545635995 shutdown_connections 2023-12-25T21:08:24.929 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.926+0000 7fa16f078700 1 --2- 172.21.15.174:0/1545635995 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa15802ffb0 0x7fa158032470 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:24.929 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.926+0000 7fa16f078700 1 --2- 172.21.15.174:0/1545635995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa168060200 0x7fa1681088d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:24.929 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.926+0000 7fa16f078700 1 -- 172.21.15.174:0/1545635995 >> 172.21.15.174:0/1545635995 conn(0x7fa1680f0b70 msgr2=0x7fa1680f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:24.929 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.926+0000 7fa16f078700 1 -- 172.21.15.174:0/1545635995 shutdown_connections 2023-12-25T21:08:24.929 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:24.926+0000 7fa16f078700 1 -- 172.21.15.174:0/1545635995 wait complete. 2023-12-25T21:08:24.929 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:25.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:25 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:24.925944+0000 mon.smithi061 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.174:0/1545635995' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:26.327 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:26.327 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:26.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:26 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:24.580013+0000 mgr.smithi061.jxxeke (mgr.14164) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:27.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:27 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:26.580285+0000 mgr.smithi061.jxxeke (mgr.14164) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:27.756 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.754+0000 7f713158f700 1 -- 172.21.15.174:0/1732699082 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f712c0f6510 msgr2=0x7f712c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:27.756 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.754+0000 7f713158f700 1 --2- 172.21.15.174:0/1732699082 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f712c0f6510 0x7f712c0f6930 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f7120009fb0 tx=0x7f7120009b30).stop 2023-12-25T21:08:27.756 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.754+0000 7f713158f700 1 -- 172.21.15.174:0/1732699082 shutdown_connections 2023-12-25T21:08:27.756 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.754+0000 7f713158f700 1 --2- 172.21.15.174:0/1732699082 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f712c0f6510 0x7f712c0f6930 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:27.756 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.754+0000 7f713158f700 1 -- 172.21.15.174:0/1732699082 >> 172.21.15.174:0/1732699082 conn(0x7f712c0f1c70 msgr2=0x7f712c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:27.756 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.754+0000 7f713158f700 1 -- 172.21.15.174:0/1732699082 shutdown_connections 2023-12-25T21:08:27.756 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.754+0000 7f713158f700 1 -- 172.21.15.174:0/1732699082 wait complete. 2023-12-25T21:08:27.756 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.755+0000 7f713158f700 1 Processor -- start 2023-12-25T21:08:27.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.755+0000 7f713158f700 1 -- start start 2023-12-25T21:08:27.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.755+0000 7f713158f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f712c0f6510 0x7f712c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:27.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.755+0000 7f713158f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c106c20 con 0x7f712c0f6510 2023-12-25T21:08:27.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.755+0000 7f712affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f712c0f6510 0x7f712c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:27.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.755+0000 7f712affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f712c0f6510 0x7f712c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:49088/0 (socket says 172.21.15.174:49088) 2023-12-25T21:08:27.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.755+0000 7f712affd700 1 -- 172.21.15.174:0/1739439765 learned_addr learned my addr 172.21.15.174:0/1739439765 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:27.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.755+0000 7f712affd700 1 -- 172.21.15.174:0/1739439765 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7120005040 con 0x7f712c0f6510 2023-12-25T21:08:27.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.755+0000 7f712affd700 1 --2- 172.21.15.174:0/1739439765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f712c0f6510 0x7f712c106730 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f7120013510 tx=0x7f712000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:27.758 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.756+0000 7f71137fe700 1 -- 172.21.15.174:0/1739439765 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7120009320 con 0x7f712c0f6510 2023-12-25T21:08:27.758 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.756+0000 7f713158f700 1 -- 172.21.15.174:0/1739439765 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f712c106e20 con 0x7f712c0f6510 2023-12-25T21:08:27.758 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.756+0000 7f71137fe700 1 -- 172.21.15.174:0/1739439765 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7120013f00 con 0x7f712c0f6510 2023-12-25T21:08:27.759 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.756+0000 7f71137fe700 1 -- 172.21.15.174:0/1739439765 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7120014d90 con 0x7f712c0f6510 2023-12-25T21:08:27.759 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.756+0000 7f713158f700 1 -- 172.21.15.174:0/1739439765 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712c1072b0 con 0x7f712c0f6510 2023-12-25T21:08:27.759 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.756+0000 7f71137fe700 1 -- 172.21.15.174:0/1739439765 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71200095f0 con 0x7f712c0f6510 2023-12-25T21:08:27.759 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.756+0000 7f713158f700 1 -- 172.21.15.174:0/1739439765 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f712c045ad0 con 0x7f712c0f6510 2023-12-25T21:08:27.759 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.757+0000 7f71137fe700 1 --2- 172.21.15.174:0/1739439765 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f711402ff60 0x7f7114032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:27.759 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.757+0000 7f71137fe700 1 -- 172.21.15.174:0/1739439765 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f71200410a0 con 0x7f712c0f6510 2023-12-25T21:08:27.759 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.757+0000 7f712a7fc700 1 --2- 172.21.15.174:0/1739439765 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f711402ff60 0x7f7114032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:27.759 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.757+0000 7f712a7fc700 1 --2- 172.21.15.174:0/1739439765 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f711402ff60 0x7f7114032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f711c0096a0 tx=0x7f711c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:27.762 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.762+0000 7f71137fe700 1 -- 172.21.15.174:0/1739439765 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f712001d7f0 con 0x7f712c0f6510 2023-12-25T21:08:27.965 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.964+0000 7f713158f700 1 -- 172.21.15.174:0/1739439765 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f712c0fbf60 con 0x7f712c0f6510 2023-12-25T21:08:27.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.965+0000 7f71137fe700 1 -- 172.21.15.174:0/1739439765 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7120010230 con 0x7f712c0f6510 2023-12-25T21:08:27.966 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:27.966 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:27.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.966+0000 7f713158f700 1 -- 172.21.15.174:0/1739439765 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f711402ff60 msgr2=0x7f7114032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:27.968 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.966+0000 7f713158f700 1 --2- 172.21.15.174:0/1739439765 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f711402ff60 0x7f7114032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f711c0096a0 tx=0x7f711c006b40).stop 2023-12-25T21:08:27.968 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.966+0000 7f713158f700 1 -- 172.21.15.174:0/1739439765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f712c0f6510 msgr2=0x7f712c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:27.968 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.966+0000 7f713158f700 1 --2- 172.21.15.174:0/1739439765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f712c0f6510 0x7f712c106730 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f7120013510 tx=0x7f712000b040).stop 2023-12-25T21:08:27.968 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.967+0000 7f713158f700 1 -- 172.21.15.174:0/1739439765 shutdown_connections 2023-12-25T21:08:27.968 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.967+0000 7f713158f700 1 --2- 172.21.15.174:0/1739439765 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f711402ff60 0x7f7114032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:27.968 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.967+0000 7f713158f700 1 --2- 172.21.15.174:0/1739439765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f712c0f6510 0x7f712c106730 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:27.968 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.967+0000 7f713158f700 1 -- 172.21.15.174:0/1739439765 >> 172.21.15.174:0/1739439765 conn(0x7f712c0f1c70 msgr2=0x7f712c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:27.968 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.967+0000 7f713158f700 1 -- 172.21.15.174:0/1739439765 shutdown_connections 2023-12-25T21:08:27.968 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:27.967+0000 7f713158f700 1 -- 172.21.15.174:0/1739439765 wait complete. 2023-12-25T21:08:27.969 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:28.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:28 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:27.966374+0000 mon.smithi061 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.174:0/1739439765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:29.374 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:29.374 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:29.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:29 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:28.580670+0000 mgr.smithi061.jxxeke (mgr.14164) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:30.486 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.484+0000 7feaa379a700 1 -- 172.21.15.174:0/777819805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea9c060280 msgr2=0x7fea9c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:30.486 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.484+0000 7feaa379a700 1 --2- 172.21.15.174:0/777819805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea9c060280 0x7fea9c0f90f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fea98009fb0 tx=0x7fea98009b30).stop 2023-12-25T21:08:30.486 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.484+0000 7feaa379a700 1 -- 172.21.15.174:0/777819805 shutdown_connections 2023-12-25T21:08:30.486 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.484+0000 7feaa379a700 1 --2- 172.21.15.174:0/777819805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea9c060280 0x7fea9c0f90f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:30.486 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.484+0000 7feaa379a700 1 -- 172.21.15.174:0/777819805 >> 172.21.15.174:0/777819805 conn(0x7fea9c0f0b70 msgr2=0x7fea9c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:30.486 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.484+0000 7feaa379a700 1 -- 172.21.15.174:0/777819805 shutdown_connections 2023-12-25T21:08:30.486 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.484+0000 7feaa379a700 1 -- 172.21.15.174:0/777819805 wait complete. 2023-12-25T21:08:30.487 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.485+0000 7feaa379a700 1 Processor -- start 2023-12-25T21:08:30.487 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.485+0000 7feaa379a700 1 -- start start 2023-12-25T21:08:30.487 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.485+0000 7feaa379a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea9c060280 0x7fea9c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:30.487 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.485+0000 7feaa379a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9c106bd0 con 0x7fea9c060280 2023-12-25T21:08:30.487 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.485+0000 7feaa1536700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea9c060280 0x7fea9c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:30.487 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.485+0000 7feaa1536700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea9c060280 0x7fea9c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:49098/0 (socket says 172.21.15.174:49098) 2023-12-25T21:08:30.487 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.485+0000 7feaa1536700 1 -- 172.21.15.174:0/2265690544 learned_addr learned my addr 172.21.15.174:0/2265690544 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:30.488 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.486+0000 7feaa1536700 1 -- 172.21.15.174:0/2265690544 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea98005040 con 0x7fea9c060280 2023-12-25T21:08:30.489 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.486+0000 7feaa1536700 1 --2- 172.21.15.174:0/2265690544 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea9c060280 0x7fea9c1066e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fea98013510 tx=0x7fea9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:30.489 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.486+0000 7fea91ffb700 1 -- 172.21.15.174:0/2265690544 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea98009320 con 0x7fea9c060280 2023-12-25T21:08:30.489 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.486+0000 7feaa379a700 1 -- 172.21.15.174:0/2265690544 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea9c106dd0 con 0x7fea9c060280 2023-12-25T21:08:30.489 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.486+0000 7fea91ffb700 1 -- 172.21.15.174:0/2265690544 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea98013f00 con 0x7fea9c060280 2023-12-25T21:08:30.489 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.486+0000 7fea91ffb700 1 -- 172.21.15.174:0/2265690544 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea98014dc0 con 0x7fea9c060280 2023-12-25T21:08:30.489 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.486+0000 7feaa379a700 1 -- 172.21.15.174:0/2265690544 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea9c107260 con 0x7fea9c060280 2023-12-25T21:08:30.489 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.487+0000 7fea91ffb700 1 -- 172.21.15.174:0/2265690544 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea980095f0 con 0x7fea9c060280 2023-12-25T21:08:30.489 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.487+0000 7feaa379a700 1 -- 172.21.15.174:0/2265690544 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea9c045ad0 con 0x7fea9c060280 2023-12-25T21:08:30.489 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.487+0000 7fea91ffb700 1 --2- 172.21.15.174:0/2265690544 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fea8802ff60 0x7fea88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:30.489 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.487+0000 7fea91ffb700 1 -- 172.21.15.174:0/2265690544 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea98041040 con 0x7fea9c060280 2023-12-25T21:08:30.489 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.488+0000 7feaa0d35700 1 --2- 172.21.15.174:0/2265690544 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fea8802ff60 0x7fea88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:30.489 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.488+0000 7feaa0d35700 1 --2- 172.21.15.174:0/2265690544 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fea8802ff60 0x7fea88032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fea8c0096a0 tx=0x7fea8c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:30.493 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.492+0000 7fea91ffb700 1 -- 172.21.15.174:0/2265690544 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea98014420 con 0x7fea9c060280 2023-12-25T21:08:30.698 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.698+0000 7feaa379a700 1 -- 172.21.15.174:0/2265690544 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea9c108110 con 0x7fea9c060280 2023-12-25T21:08:30.699 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.698+0000 7fea91ffb700 1 -- 172.21.15.174:0/2265690544 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fea98010230 con 0x7fea9c060280 2023-12-25T21:08:30.699 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:30.699 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:30.701 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.700+0000 7feaa379a700 1 -- 172.21.15.174:0/2265690544 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fea8802ff60 msgr2=0x7fea88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:30.701 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.700+0000 7feaa379a700 1 --2- 172.21.15.174:0/2265690544 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fea8802ff60 0x7fea88032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fea8c0096a0 tx=0x7fea8c006b40).stop 2023-12-25T21:08:30.701 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.700+0000 7feaa379a700 1 -- 172.21.15.174:0/2265690544 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea9c060280 msgr2=0x7fea9c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:30.701 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.700+0000 7feaa379a700 1 --2- 172.21.15.174:0/2265690544 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea9c060280 0x7fea9c1066e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fea98013510 tx=0x7fea9800b040).stop 2023-12-25T21:08:30.701 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.700+0000 7feaa379a700 1 -- 172.21.15.174:0/2265690544 shutdown_connections 2023-12-25T21:08:30.701 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.700+0000 7feaa379a700 1 --2- 172.21.15.174:0/2265690544 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fea8802ff60 0x7fea88032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:30.701 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.700+0000 7feaa379a700 1 --2- 172.21.15.174:0/2265690544 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea9c060280 0x7fea9c1066e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:30.702 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.700+0000 7feaa379a700 1 -- 172.21.15.174:0/2265690544 >> 172.21.15.174:0/2265690544 conn(0x7fea9c0f0b70 msgr2=0x7fea9c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:30.702 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.700+0000 7feaa379a700 1 -- 172.21.15.174:0/2265690544 shutdown_connections 2023-12-25T21:08:30.702 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:30.700+0000 7feaa379a700 1 -- 172.21.15.174:0/2265690544 wait complete. 2023-12-25T21:08:30.702 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:31.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:31 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:30.581080+0000 mgr.smithi061.jxxeke (mgr.14164) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:31.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:31 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:30.699694+0000 mon.smithi061 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.174:0/2265690544' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:32.095 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:32.096 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:33.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.209+0000 7effc2b1e700 1 -- 172.21.15.174:0/2491577984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7effbc0f5410 msgr2=0x7effbc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:33.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.209+0000 7effc2b1e700 1 --2- 172.21.15.174:0/2491577984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7effbc0f5410 0x7effbc0f5830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7effb0009fb0 tx=0x7effb0009b30).stop 2023-12-25T21:08:33.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.209+0000 7effc2b1e700 1 -- 172.21.15.174:0/2491577984 shutdown_connections 2023-12-25T21:08:33.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.209+0000 7effc2b1e700 1 --2- 172.21.15.174:0/2491577984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7effbc0f5410 0x7effbc0f5830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:33.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.209+0000 7effc2b1e700 1 -- 172.21.15.174:0/2491577984 >> 172.21.15.174:0/2491577984 conn(0x7effbc0f0b70 msgr2=0x7effbc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:33.211 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.209+0000 7effc2b1e700 1 -- 172.21.15.174:0/2491577984 shutdown_connections 2023-12-25T21:08:33.211 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.209+0000 7effc2b1e700 1 -- 172.21.15.174:0/2491577984 wait complete. 2023-12-25T21:08:33.211 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.209+0000 7effc2b1e700 1 Processor -- start 2023-12-25T21:08:33.211 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.210+0000 7effc2b1e700 1 -- start start 2023-12-25T21:08:33.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.210+0000 7effc2b1e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7effbc0f5410 0x7effbc1055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:33.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.210+0000 7effc2b1e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effbc105ac0 con 0x7effbc0f5410 2023-12-25T21:08:33.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.210+0000 7effc08ba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7effbc0f5410 0x7effbc1055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:33.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.210+0000 7effc08ba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7effbc0f5410 0x7effbc1055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:49106/0 (socket says 172.21.15.174:49106) 2023-12-25T21:08:33.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.210+0000 7effc08ba700 1 -- 172.21.15.174:0/230973426 learned_addr learned my addr 172.21.15.174:0/230973426 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:33.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.210+0000 7effc08ba700 1 -- 172.21.15.174:0/230973426 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb0005040 con 0x7effbc0f5410 2023-12-25T21:08:33.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.210+0000 7effc08ba700 1 --2- 172.21.15.174:0/230973426 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7effbc0f5410 0x7effbc1055d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7effb0014000 tx=0x7effb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:33.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.211+0000 7effb97fa700 1 -- 172.21.15.174:0/230973426 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effb00095f0 con 0x7effbc0f5410 2023-12-25T21:08:33.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.211+0000 7effc2b1e700 1 -- 172.21.15.174:0/230973426 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effbc105cc0 con 0x7effbc0f5410 2023-12-25T21:08:33.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.211+0000 7effb97fa700 1 -- 172.21.15.174:0/230973426 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7effb0007e70 con 0x7effbc0f5410 2023-12-25T21:08:33.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.211+0000 7effb97fa700 1 -- 172.21.15.174:0/230973426 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effb0014ea0 con 0x7effbc0f5410 2023-12-25T21:08:33.214 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.211+0000 7effc2b1e700 1 -- 172.21.15.174:0/230973426 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effbc106150 con 0x7effbc0f5410 2023-12-25T21:08:33.214 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.211+0000 7effb97fa700 1 -- 172.21.15.174:0/230973426 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7effb0014690 con 0x7effbc0f5410 2023-12-25T21:08:33.214 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.212+0000 7effc2b1e700 1 -- 172.21.15.174:0/230973426 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effbc045ad0 con 0x7effbc0f5410 2023-12-25T21:08:33.214 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.212+0000 7effb97fa700 1 --2- 172.21.15.174:0/230973426 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7effac02ff60 0x7effac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:33.214 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.212+0000 7effb97fa700 1 -- 172.21.15.174:0/230973426 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7effb0040870 con 0x7effbc0f5410 2023-12-25T21:08:33.214 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.212+0000 7effbbfff700 1 --2- 172.21.15.174:0/230973426 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7effac02ff60 0x7effac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:33.214 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.212+0000 7effbbfff700 1 --2- 172.21.15.174:0/230973426 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7effac02ff60 0x7effac032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7effa80096a0 tx=0x7effa8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:33.217 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.217+0000 7effb97fa700 1 -- 172.21.15.174:0/230973426 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7effb000f300 con 0x7effbc0f5410 2023-12-25T21:08:33.423 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.422+0000 7effc2b1e700 1 -- 172.21.15.174:0/230973426 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7effbc107000 con 0x7effbc0f5410 2023-12-25T21:08:33.424 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.423+0000 7effb97fa700 1 -- 172.21.15.174:0/230973426 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7effb000f300 con 0x7effbc0f5410 2023-12-25T21:08:33.424 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:33.424 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:33.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.425+0000 7effc2b1e700 1 -- 172.21.15.174:0/230973426 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7effac02ff60 msgr2=0x7effac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:33.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.425+0000 7effc2b1e700 1 --2- 172.21.15.174:0/230973426 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7effac02ff60 0x7effac032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7effa80096a0 tx=0x7effa8006b40).stop 2023-12-25T21:08:33.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.425+0000 7effc2b1e700 1 -- 172.21.15.174:0/230973426 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7effbc0f5410 msgr2=0x7effbc1055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:33.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.425+0000 7effc2b1e700 1 --2- 172.21.15.174:0/230973426 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7effbc0f5410 0x7effbc1055d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7effb0014000 tx=0x7effb000b040).stop 2023-12-25T21:08:33.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.425+0000 7effc2b1e700 1 -- 172.21.15.174:0/230973426 shutdown_connections 2023-12-25T21:08:33.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.425+0000 7effc2b1e700 1 --2- 172.21.15.174:0/230973426 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7effac02ff60 0x7effac032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:33.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.425+0000 7effc2b1e700 1 --2- 172.21.15.174:0/230973426 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7effbc0f5410 0x7effbc1055d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:33.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.425+0000 7effc2b1e700 1 -- 172.21.15.174:0/230973426 >> 172.21.15.174:0/230973426 conn(0x7effbc0f0b70 msgr2=0x7effbc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:33.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.425+0000 7effc2b1e700 1 -- 172.21.15.174:0/230973426 shutdown_connections 2023-12-25T21:08:33.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:33.425+0000 7effc2b1e700 1 -- 172.21.15.174:0/230973426 wait complete. 2023-12-25T21:08:33.427 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:33.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:33 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:32.581432+0000 mgr.smithi061.jxxeke (mgr.14164) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:33.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:33 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:33.424359+0000 mon.smithi061 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.174:0/230973426' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:34.830 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:34.831 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:35.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:35 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:34.581842+0000 mgr.smithi061.jxxeke (mgr.14164) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:36.017 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.015+0000 7fab435a2700 1 -- 172.21.15.174:0/3343520314 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab3c0f5410 msgr2=0x7fab3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:36.017 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.015+0000 7fab435a2700 1 --2- 172.21.15.174:0/3343520314 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab3c0f5410 0x7fab3c0f5830 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fab2c009fb0 tx=0x7fab2c009b30).stop 2023-12-25T21:08:36.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.015+0000 7fab435a2700 1 -- 172.21.15.174:0/3343520314 shutdown_connections 2023-12-25T21:08:36.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.015+0000 7fab435a2700 1 --2- 172.21.15.174:0/3343520314 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab3c0f5410 0x7fab3c0f5830 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:36.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.015+0000 7fab435a2700 1 -- 172.21.15.174:0/3343520314 >> 172.21.15.174:0/3343520314 conn(0x7fab3c0f0b70 msgr2=0x7fab3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:36.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.015+0000 7fab435a2700 1 -- 172.21.15.174:0/3343520314 shutdown_connections 2023-12-25T21:08:36.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.015+0000 7fab435a2700 1 -- 172.21.15.174:0/3343520314 wait complete. 2023-12-25T21:08:36.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.015+0000 7fab435a2700 1 Processor -- start 2023-12-25T21:08:36.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.015+0000 7fab435a2700 1 -- start start 2023-12-25T21:08:36.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.015+0000 7fab435a2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab3c0f5410 0x7fab3c1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:36.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.015+0000 7fab435a2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c060280 con 0x7fab3c0f5410 2023-12-25T21:08:36.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.016+0000 7fab4133e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab3c0f5410 0x7fab3c1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:36.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.016+0000 7fab4133e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab3c0f5410 0x7fab3c1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:49112/0 (socket says 172.21.15.174:49112) 2023-12-25T21:08:36.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.016+0000 7fab4133e700 1 -- 172.21.15.174:0/173443862 learned_addr learned my addr 172.21.15.174:0/173443862 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:36.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.016+0000 7fab4133e700 1 -- 172.21.15.174:0/173443862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2c005040 con 0x7fab3c0f5410 2023-12-25T21:08:36.019 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.016+0000 7fab4133e700 1 --2- 172.21.15.174:0/173443862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab3c0f5410 0x7fab3c1093f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fab2c014000 tx=0x7fab2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:36.019 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.016+0000 7fab31ffb700 1 -- 172.21.15.174:0/173443862 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab2c0095f0 con 0x7fab3c0f5410 2023-12-25T21:08:36.019 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.016+0000 7fab435a2700 1 -- 172.21.15.174:0/173443862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab3c060480 con 0x7fab3c0f5410 2023-12-25T21:08:36.019 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.016+0000 7fab31ffb700 1 -- 172.21.15.174:0/173443862 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab2c007e70 con 0x7fab3c0f5410 2023-12-25T21:08:36.019 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.016+0000 7fab31ffb700 1 -- 172.21.15.174:0/173443862 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab2c014ea0 con 0x7fab3c0f5410 2023-12-25T21:08:36.019 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.017+0000 7fab435a2700 1 -- 172.21.15.174:0/173443862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3c109b20 con 0x7fab3c0f5410 2023-12-25T21:08:36.020 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.017+0000 7fab435a2700 1 -- 172.21.15.174:0/173443862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab3c045ad0 con 0x7fab3c0f5410 2023-12-25T21:08:36.020 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.018+0000 7fab31ffb700 1 -- 172.21.15.174:0/173443862 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fab2c025070 con 0x7fab3c0f5410 2023-12-25T21:08:36.020 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.018+0000 7fab31ffb700 1 --2- 172.21.15.174:0/173443862 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fab2802ff60 0x7fab28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:36.020 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.018+0000 7fab31ffb700 1 -- 172.21.15.174:0/173443862 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fab2c040940 con 0x7fab3c0f5410 2023-12-25T21:08:36.020 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.018+0000 7fab40b3d700 1 --2- 172.21.15.174:0/173443862 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fab2802ff60 0x7fab28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:36.020 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.018+0000 7fab40b3d700 1 --2- 172.21.15.174:0/173443862 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fab2802ff60 0x7fab28032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fab380096a0 tx=0x7fab38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:36.023 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.022+0000 7fab31ffb700 1 -- 172.21.15.174:0/173443862 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab2c0079c0 con 0x7fab3c0f5410 2023-12-25T21:08:36.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.226+0000 7fab435a2700 1 -- 172.21.15.174:0/173443862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab3c10a370 con 0x7fab3c0f5410 2023-12-25T21:08:36.228 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.227+0000 7fab31ffb700 1 -- 172.21.15.174:0/173443862 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fab2c00f300 con 0x7fab3c0f5410 2023-12-25T21:08:36.228 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:36.228 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:36.229 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.228+0000 7fab435a2700 1 -- 172.21.15.174:0/173443862 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fab2802ff60 msgr2=0x7fab28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:36.230 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.229+0000 7fab435a2700 1 --2- 172.21.15.174:0/173443862 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fab2802ff60 0x7fab28032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fab380096a0 tx=0x7fab38006b40).stop 2023-12-25T21:08:36.230 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.229+0000 7fab435a2700 1 -- 172.21.15.174:0/173443862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab3c0f5410 msgr2=0x7fab3c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:36.230 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.229+0000 7fab435a2700 1 --2- 172.21.15.174:0/173443862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab3c0f5410 0x7fab3c1093f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fab2c014000 tx=0x7fab2c00b040).stop 2023-12-25T21:08:36.230 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.229+0000 7fab435a2700 1 -- 172.21.15.174:0/173443862 shutdown_connections 2023-12-25T21:08:36.230 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.229+0000 7fab435a2700 1 --2- 172.21.15.174:0/173443862 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fab2802ff60 0x7fab28032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:36.230 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.229+0000 7fab435a2700 1 --2- 172.21.15.174:0/173443862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab3c0f5410 0x7fab3c1093f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:36.230 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.229+0000 7fab435a2700 1 -- 172.21.15.174:0/173443862 >> 172.21.15.174:0/173443862 conn(0x7fab3c0f0b70 msgr2=0x7fab3c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:36.230 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.229+0000 7fab435a2700 1 -- 172.21.15.174:0/173443862 shutdown_connections 2023-12-25T21:08:36.230 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:36.229+0000 7fab435a2700 1 -- 172.21.15.174:0/173443862 wait complete. 2023-12-25T21:08:36.231 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:36.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:36 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:36.228475+0000 mon.smithi061 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.174:0/173443862' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:37.632 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:37.632 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:38.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:38 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:36.582257+0000 mgr.smithi061.jxxeke (mgr.14164) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:38.862 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.860+0000 7f8448695700 1 -- 172.21.15.174:0/3922181070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84400f6510 msgr2=0x7f84400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:38.863 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.860+0000 7f8448695700 1 --2- 172.21.15.174:0/3922181070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84400f6510 0x7f84400f6930 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f843c009fb0 tx=0x7f843c009b30).stop 2023-12-25T21:08:38.863 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.860+0000 7f8448695700 1 -- 172.21.15.174:0/3922181070 shutdown_connections 2023-12-25T21:08:38.863 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.860+0000 7f8448695700 1 --2- 172.21.15.174:0/3922181070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84400f6510 0x7f84400f6930 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:38.863 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.860+0000 7f8448695700 1 -- 172.21.15.174:0/3922181070 >> 172.21.15.174:0/3922181070 conn(0x7f84400f1c70 msgr2=0x7f84400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:38.863 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.860+0000 7f8448695700 1 -- 172.21.15.174:0/3922181070 shutdown_connections 2023-12-25T21:08:38.863 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.860+0000 7f8448695700 1 -- 172.21.15.174:0/3922181070 wait complete. 2023-12-25T21:08:38.863 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.861+0000 7f8448695700 1 Processor -- start 2023-12-25T21:08:38.863 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.861+0000 7f8448695700 1 -- start start 2023-12-25T21:08:38.863 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.861+0000 7f8448695700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84400f6510 0x7f844006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:38.863 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.861+0000 7f8448695700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8440069e20 con 0x7f84400f6510 2023-12-25T21:08:38.863 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.861+0000 7f8446431700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84400f6510 0x7f844006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:38.863 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.861+0000 7f8446431700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84400f6510 0x7f844006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:59846/0 (socket says 172.21.15.174:59846) 2023-12-25T21:08:38.864 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.861+0000 7f8446431700 1 -- 172.21.15.174:0/3946633018 learned_addr learned my addr 172.21.15.174:0/3946633018 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:38.864 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.861+0000 7f8446431700 1 -- 172.21.15.174:0/3946633018 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c005040 con 0x7f84400f6510 2023-12-25T21:08:38.864 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.861+0000 7f8446431700 1 --2- 172.21.15.174:0/3946633018 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84400f6510 0x7f844006b9a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f843c013510 tx=0x7f843c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:38.864 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.862+0000 7f8436ffd700 1 -- 172.21.15.174:0/3946633018 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f843c009320 con 0x7f84400f6510 2023-12-25T21:08:38.864 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.862+0000 7f8436ffd700 1 -- 172.21.15.174:0/3946633018 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f843c013f00 con 0x7f84400f6510 2023-12-25T21:08:38.864 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.862+0000 7f8448695700 1 -- 172.21.15.174:0/3946633018 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8440069fc0 con 0x7f84400f6510 2023-12-25T21:08:38.864 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.862+0000 7f8436ffd700 1 -- 172.21.15.174:0/3946633018 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f843c014d90 con 0x7f84400f6510 2023-12-25T21:08:38.864 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.862+0000 7f8448695700 1 -- 172.21.15.174:0/3946633018 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844006a450 con 0x7f84400f6510 2023-12-25T21:08:38.865 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.862+0000 7f8436ffd700 1 -- 172.21.15.174:0/3946633018 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f843c0095f0 con 0x7f84400f6510 2023-12-25T21:08:38.865 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.862+0000 7f8448695700 1 -- 172.21.15.174:0/3946633018 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8440059100 con 0x7f84400f6510 2023-12-25T21:08:38.865 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.863+0000 7f8436ffd700 1 --2- 172.21.15.174:0/3946633018 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f842c02ffb0 0x7f842c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:38.865 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.863+0000 7f8436ffd700 1 -- 172.21.15.174:0/3946633018 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f843c040fe0 con 0x7f84400f6510 2023-12-25T21:08:38.865 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.863+0000 7f8445c30700 1 --2- 172.21.15.174:0/3946633018 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f842c02ffb0 0x7f842c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:38.865 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.863+0000 7f8445c30700 1 --2- 172.21.15.174:0/3946633018 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f842c02ffb0 0x7f842c032470 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f84300096a0 tx=0x7f8430006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:38.868 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:38.867+0000 7f8436ffd700 1 -- 172.21.15.174:0/3946633018 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f843c06d2b0 con 0x7f84400f6510 2023-12-25T21:08:39.074 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:39.073+0000 7f8448695700 1 -- 172.21.15.174:0/3946633018 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84400fbf60 con 0x7f84400f6510 2023-12-25T21:08:39.074 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:39.074+0000 7f8436ffd700 1 -- 172.21.15.174:0/3946633018 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f843c014420 con 0x7f84400f6510 2023-12-25T21:08:39.074 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:39.074 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:39.076 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:39.075+0000 7f8448695700 1 -- 172.21.15.174:0/3946633018 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f842c02ffb0 msgr2=0x7f842c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:39.077 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:39.075+0000 7f8448695700 1 --2- 172.21.15.174:0/3946633018 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f842c02ffb0 0x7f842c032470 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f84300096a0 tx=0x7f8430006b40).stop 2023-12-25T21:08:39.077 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:39.075+0000 7f8448695700 1 -- 172.21.15.174:0/3946633018 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84400f6510 msgr2=0x7f844006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:39.077 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:39.075+0000 7f8448695700 1 --2- 172.21.15.174:0/3946633018 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84400f6510 0x7f844006b9a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f843c013510 tx=0x7f843c00b040).stop 2023-12-25T21:08:39.077 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:39.075+0000 7f8448695700 1 -- 172.21.15.174:0/3946633018 shutdown_connections 2023-12-25T21:08:39.077 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:39.075+0000 7f8448695700 1 --2- 172.21.15.174:0/3946633018 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f842c02ffb0 0x7f842c032470 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:39.077 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:39.075+0000 7f8448695700 1 --2- 172.21.15.174:0/3946633018 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84400f6510 0x7f844006b9a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:39.077 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:39.075+0000 7f8448695700 1 -- 172.21.15.174:0/3946633018 >> 172.21.15.174:0/3946633018 conn(0x7f84400f1c70 msgr2=0x7f84400f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:39.077 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:39.075+0000 7f8448695700 1 -- 172.21.15.174:0/3946633018 shutdown_connections 2023-12-25T21:08:39.077 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:39.075+0000 7f8448695700 1 -- 172.21.15.174:0/3946633018 wait complete. 2023-12-25T21:08:39.077 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:39.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:39 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:39.074962+0000 mon.smithi061 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.174:0/3946633018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:40.507 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:40.507 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:40.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:40 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:38.582558+0000 mgr.smithi061.jxxeke (mgr.14164) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:41.711 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.709+0000 7efecb651700 1 -- 172.21.15.174:0/2679767234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efec4060280 msgr2=0x7efec40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:41.711 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.709+0000 7efecb651700 1 --2- 172.21.15.174:0/2679767234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efec4060280 0x7efec40f90f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7efec0009fb0 tx=0x7efec0009b30).stop 2023-12-25T21:08:41.711 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.709+0000 7efecb651700 1 -- 172.21.15.174:0/2679767234 shutdown_connections 2023-12-25T21:08:41.711 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.709+0000 7efecb651700 1 --2- 172.21.15.174:0/2679767234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efec4060280 0x7efec40f90f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:41.711 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.709+0000 7efecb651700 1 -- 172.21.15.174:0/2679767234 >> 172.21.15.174:0/2679767234 conn(0x7efec40f0b70 msgr2=0x7efec40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:41.711 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.709+0000 7efecb651700 1 -- 172.21.15.174:0/2679767234 shutdown_connections 2023-12-25T21:08:41.711 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.710+0000 7efecb651700 1 -- 172.21.15.174:0/2679767234 wait complete. 2023-12-25T21:08:41.713 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.710+0000 7efecb651700 1 Processor -- start 2023-12-25T21:08:41.713 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.711+0000 7efecb651700 1 -- start start 2023-12-25T21:08:41.713 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.711+0000 7efecb651700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efec4060280 0x7efec40f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:41.713 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.711+0000 7efecb651700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec40f4ff0 con 0x7efec4060280 2023-12-25T21:08:41.713 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.711+0000 7efec93ed700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efec4060280 0x7efec40f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:41.713 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.711+0000 7efec93ed700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efec4060280 0x7efec40f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:59870/0 (socket says 172.21.15.174:59870) 2023-12-25T21:08:41.713 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.711+0000 7efec93ed700 1 -- 172.21.15.174:0/3695027093 learned_addr learned my addr 172.21.15.174:0/3695027093 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:41.713 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.711+0000 7efec93ed700 1 -- 172.21.15.174:0/3695027093 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec0005040 con 0x7efec4060280 2023-12-25T21:08:41.713 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.711+0000 7efec93ed700 1 --2- 172.21.15.174:0/3695027093 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efec4060280 0x7efec40f6b70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7efec0013510 tx=0x7efec000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:41.713 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.712+0000 7efeb9ffb700 1 -- 172.21.15.174:0/3695027093 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efec0009320 con 0x7efec4060280 2023-12-25T21:08:41.713 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.712+0000 7efecb651700 1 -- 172.21.15.174:0/3695027093 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec40f5190 con 0x7efec4060280 2023-12-25T21:08:41.713 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.712+0000 7efeb9ffb700 1 -- 172.21.15.174:0/3695027093 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efec0013f00 con 0x7efec4060280 2023-12-25T21:08:41.714 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.712+0000 7efeb9ffb700 1 -- 172.21.15.174:0/3695027093 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efec0014dc0 con 0x7efec4060280 2023-12-25T21:08:41.714 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.712+0000 7efecb651700 1 -- 172.21.15.174:0/3695027093 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec40f5620 con 0x7efec4060280 2023-12-25T21:08:41.714 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.713+0000 7efeb9ffb700 1 -- 172.21.15.174:0/3695027093 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efec0014420 con 0x7efec4060280 2023-12-25T21:08:41.714 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.713+0000 7efecb651700 1 -- 172.21.15.174:0/3695027093 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec4059100 con 0x7efec4060280 2023-12-25T21:08:41.715 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.713+0000 7efeb9ffb700 1 --2- 172.21.15.174:0/3695027093 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efeb002ffb0 0x7efeb0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:41.715 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.713+0000 7efeb9ffb700 1 -- 172.21.15.174:0/3695027093 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efec00410b0 con 0x7efec4060280 2023-12-25T21:08:41.715 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.713+0000 7efec8bec700 1 --2- 172.21.15.174:0/3695027093 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efeb002ffb0 0x7efeb0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:41.715 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.713+0000 7efec8bec700 1 --2- 172.21.15.174:0/3695027093 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efeb002ffb0 0x7efeb0032470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7efeb40096a0 tx=0x7efeb4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:41.718 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.718+0000 7efeb9ffb700 1 -- 172.21.15.174:0/3695027093 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efec006d2b0 con 0x7efec4060280 2023-12-25T21:08:41.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.924+0000 7efecb651700 1 -- 172.21.15.174:0/3695027093 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efec40fbf50 con 0x7efec4060280 2023-12-25T21:08:41.925 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.925+0000 7efeb9ffb700 1 -- 172.21.15.174:0/3695027093 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efec00095f0 con 0x7efec4060280 2023-12-25T21:08:41.926 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:41.926 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:41.927 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.926+0000 7efecb651700 1 -- 172.21.15.174:0/3695027093 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efeb002ffb0 msgr2=0x7efeb0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:41.927 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.926+0000 7efecb651700 1 --2- 172.21.15.174:0/3695027093 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efeb002ffb0 0x7efeb0032470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7efeb40096a0 tx=0x7efeb4006b40).stop 2023-12-25T21:08:41.927 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.926+0000 7efecb651700 1 -- 172.21.15.174:0/3695027093 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efec4060280 msgr2=0x7efec40f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:41.927 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.926+0000 7efecb651700 1 --2- 172.21.15.174:0/3695027093 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efec4060280 0x7efec40f6b70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7efec0013510 tx=0x7efec000b040).stop 2023-12-25T21:08:41.927 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.926+0000 7efecb651700 1 -- 172.21.15.174:0/3695027093 shutdown_connections 2023-12-25T21:08:41.928 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.926+0000 7efecb651700 1 --2- 172.21.15.174:0/3695027093 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efeb002ffb0 0x7efeb0032470 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:41.928 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.926+0000 7efecb651700 1 --2- 172.21.15.174:0/3695027093 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efec4060280 0x7efec40f6b70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:41.928 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.926+0000 7efecb651700 1 -- 172.21.15.174:0/3695027093 >> 172.21.15.174:0/3695027093 conn(0x7efec40f0b70 msgr2=0x7efec40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:41.928 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.926+0000 7efecb651700 1 -- 172.21.15.174:0/3695027093 shutdown_connections 2023-12-25T21:08:41.928 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:41.926+0000 7efecb651700 1 -- 172.21.15.174:0/3695027093 wait complete. 2023-12-25T21:08:41.928 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:42.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:42 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:40.582920+0000 mgr.smithi061.jxxeke (mgr.14164) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:42.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:42 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:41.925815+0000 mon.smithi061 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.174:0/3695027093' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:43.324 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:43.325 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:44.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.468+0000 7f9227d52700 1 -- 172.21.15.174:0/4034748038 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92200f6510 msgr2=0x7f92200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:44.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.468+0000 7f9227d52700 1 --2- 172.21.15.174:0/4034748038 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92200f6510 0x7f92200f6930 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f9210009fb0 tx=0x7f9210009b30).stop 2023-12-25T21:08:44.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.469+0000 7f9227d52700 1 -- 172.21.15.174:0/4034748038 shutdown_connections 2023-12-25T21:08:44.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.469+0000 7f9227d52700 1 --2- 172.21.15.174:0/4034748038 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92200f6510 0x7f92200f6930 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:44.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.469+0000 7f9227d52700 1 -- 172.21.15.174:0/4034748038 >> 172.21.15.174:0/4034748038 conn(0x7f92200f1c70 msgr2=0x7f92200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:44.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.469+0000 7f9227d52700 1 -- 172.21.15.174:0/4034748038 shutdown_connections 2023-12-25T21:08:44.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.469+0000 7f9227d52700 1 -- 172.21.15.174:0/4034748038 wait complete. 2023-12-25T21:08:44.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.469+0000 7f9227d52700 1 Processor -- start 2023-12-25T21:08:44.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.469+0000 7f9227d52700 1 -- start start 2023-12-25T21:08:44.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.469+0000 7f9227d52700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92200f6510 0x7f9220106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:44.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.469+0000 7f9227d52700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9220106c20 con 0x7f92200f6510 2023-12-25T21:08:44.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.469+0000 7f9225aee700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92200f6510 0x7f9220106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:44.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.470+0000 7f9225aee700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92200f6510 0x7f9220106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:59894/0 (socket says 172.21.15.174:59894) 2023-12-25T21:08:44.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.470+0000 7f9225aee700 1 -- 172.21.15.174:0/2494311505 learned_addr learned my addr 172.21.15.174:0/2494311505 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:44.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.470+0000 7f9225aee700 1 -- 172.21.15.174:0/2494311505 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9210005040 con 0x7f92200f6510 2023-12-25T21:08:44.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.470+0000 7f9225aee700 1 --2- 172.21.15.174:0/2494311505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92200f6510 0x7f9220106730 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f9210013510 tx=0x7f921000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:44.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.470+0000 7f92167fc700 1 -- 172.21.15.174:0/2494311505 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9210009320 con 0x7f92200f6510 2023-12-25T21:08:44.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.470+0000 7f9227d52700 1 -- 172.21.15.174:0/2494311505 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9220106e20 con 0x7f92200f6510 2023-12-25T21:08:44.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.470+0000 7f92167fc700 1 -- 172.21.15.174:0/2494311505 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9210013f00 con 0x7f92200f6510 2023-12-25T21:08:44.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.470+0000 7f92167fc700 1 -- 172.21.15.174:0/2494311505 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9210014d90 con 0x7f92200f6510 2023-12-25T21:08:44.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.470+0000 7f9227d52700 1 -- 172.21.15.174:0/2494311505 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92201072b0 con 0x7f92200f6510 2023-12-25T21:08:44.473 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.471+0000 7f92167fc700 1 -- 172.21.15.174:0/2494311505 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f92100095f0 con 0x7f92200f6510 2023-12-25T21:08:44.473 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.471+0000 7f9227d52700 1 -- 172.21.15.174:0/2494311505 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9220045ad0 con 0x7f92200f6510 2023-12-25T21:08:44.474 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.471+0000 7f92167fc700 1 --2- 172.21.15.174:0/2494311505 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f920c02ff60 0x7f920c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:44.474 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.471+0000 7f92167fc700 1 -- 172.21.15.174:0/2494311505 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9210040fc0 con 0x7f92200f6510 2023-12-25T21:08:44.474 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.471+0000 7f92252ed700 1 --2- 172.21.15.174:0/2494311505 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f920c02ff60 0x7f920c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:44.474 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.472+0000 7f92252ed700 1 --2- 172.21.15.174:0/2494311505 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f920c02ff60 0x7f920c032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f921c0096a0 tx=0x7f921c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:44.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.477+0000 7f92167fc700 1 -- 172.21.15.174:0/2494311505 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9210014420 con 0x7f92200f6510 2023-12-25T21:08:44.684 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.682+0000 7f9227d52700 1 -- 172.21.15.174:0/2494311505 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92200fbf60 con 0x7f92200f6510 2023-12-25T21:08:44.684 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.683+0000 7f92167fc700 1 -- 172.21.15.174:0/2494311505 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9210010230 con 0x7f92200f6510 2023-12-25T21:08:44.684 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:44.684 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:44.685 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.684+0000 7f9227d52700 1 -- 172.21.15.174:0/2494311505 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f920c02ff60 msgr2=0x7f920c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:44.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.684+0000 7f9227d52700 1 --2- 172.21.15.174:0/2494311505 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f920c02ff60 0x7f920c032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f921c0096a0 tx=0x7f921c006b40).stop 2023-12-25T21:08:44.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.684+0000 7f9227d52700 1 -- 172.21.15.174:0/2494311505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92200f6510 msgr2=0x7f9220106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:44.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.684+0000 7f9227d52700 1 --2- 172.21.15.174:0/2494311505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92200f6510 0x7f9220106730 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f9210013510 tx=0x7f921000b040).stop 2023-12-25T21:08:44.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.684+0000 7f9227d52700 1 -- 172.21.15.174:0/2494311505 shutdown_connections 2023-12-25T21:08:44.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.684+0000 7f9227d52700 1 --2- 172.21.15.174:0/2494311505 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f920c02ff60 0x7f920c032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:44.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.684+0000 7f9227d52700 1 --2- 172.21.15.174:0/2494311505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92200f6510 0x7f9220106730 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:44.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.684+0000 7f9227d52700 1 -- 172.21.15.174:0/2494311505 >> 172.21.15.174:0/2494311505 conn(0x7f92200f1c70 msgr2=0x7f92200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:44.687 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.685+0000 7f9227d52700 1 -- 172.21.15.174:0/2494311505 shutdown_connections 2023-12-25T21:08:44.687 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:44.685+0000 7f9227d52700 1 -- 172.21.15.174:0/2494311505 wait complete. 2023-12-25T21:08:44.687 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:44.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:44 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:42.583150+0000 mgr.smithi061.jxxeke (mgr.14164) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:45.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:45 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:44.684170+0000 mon.smithi061 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.174:0/2494311505' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:46.071 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:46.071 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:46.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:46 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:44.583538+0000 mgr.smithi061.jxxeke (mgr.14164) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:47.222 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.222+0000 7f9f0267a700 1 -- 172.21.15.174:0/1503383999 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9efc060280 msgr2=0x7f9efc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:47.222 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.222+0000 7f9f0267a700 1 --2- 172.21.15.174:0/1503383999 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9efc060280 0x7f9efc0f90f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f9eec009fb0 tx=0x7f9eec009b30).stop 2023-12-25T21:08:47.222 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.222+0000 7f9f0267a700 1 -- 172.21.15.174:0/1503383999 shutdown_connections 2023-12-25T21:08:47.222 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.222+0000 7f9f0267a700 1 --2- 172.21.15.174:0/1503383999 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9efc060280 0x7f9efc0f90f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:47.222 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.222+0000 7f9f0267a700 1 -- 172.21.15.174:0/1503383999 >> 172.21.15.174:0/1503383999 conn(0x7f9efc0f0b70 msgr2=0x7f9efc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:47.222 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.222+0000 7f9f0267a700 1 -- 172.21.15.174:0/1503383999 shutdown_connections 2023-12-25T21:08:47.222 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.222+0000 7f9f0267a700 1 -- 172.21.15.174:0/1503383999 wait complete. 2023-12-25T21:08:47.222 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.222+0000 7f9f0267a700 1 Processor -- start 2023-12-25T21:08:47.224 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.223+0000 7f9f0267a700 1 -- start start 2023-12-25T21:08:47.224 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.223+0000 7f9f0267a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9efc060280 0x7f9efc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:47.224 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.223+0000 7f9f0267a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc106bd0 con 0x7f9efc060280 2023-12-25T21:08:47.224 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.223+0000 7f9efbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9efc060280 0x7f9efc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:47.224 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.223+0000 7f9efbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9efc060280 0x7f9efc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:51836/0 (socket says 172.21.15.174:51836) 2023-12-25T21:08:47.224 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.223+0000 7f9efbfff700 1 -- 172.21.15.174:0/1364927652 learned_addr learned my addr 172.21.15.174:0/1364927652 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:47.224 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.224+0000 7f9efbfff700 1 -- 172.21.15.174:0/1364927652 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eec005040 con 0x7f9efc060280 2023-12-25T21:08:47.224 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.224+0000 7f9efbfff700 1 --2- 172.21.15.174:0/1364927652 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9efc060280 0x7f9efc1066e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f9eec013510 tx=0x7f9eec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:47.224 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.224+0000 7f9ef8ff9700 1 -- 172.21.15.174:0/1364927652 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9eec009320 con 0x7f9efc060280 2023-12-25T21:08:47.224 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.224+0000 7f9f0267a700 1 -- 172.21.15.174:0/1364927652 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9efc106dd0 con 0x7f9efc060280 2023-12-25T21:08:47.224 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.224+0000 7f9ef8ff9700 1 -- 172.21.15.174:0/1364927652 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9eec013f00 con 0x7f9efc060280 2023-12-25T21:08:47.225 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.224+0000 7f9ef8ff9700 1 -- 172.21.15.174:0/1364927652 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9eec014dc0 con 0x7f9efc060280 2023-12-25T21:08:47.225 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.224+0000 7f9f0267a700 1 -- 172.21.15.174:0/1364927652 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9efc107260 con 0x7f9efc060280 2023-12-25T21:08:47.226 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.225+0000 7f9ef8ff9700 1 -- 172.21.15.174:0/1364927652 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9eec0095f0 con 0x7f9efc060280 2023-12-25T21:08:47.226 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.225+0000 7f9f0267a700 1 -- 172.21.15.174:0/1364927652 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9efc045ad0 con 0x7f9efc060280 2023-12-25T21:08:47.226 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.225+0000 7f9ef8ff9700 1 --2- 172.21.15.174:0/1364927652 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f9ee802ff60 0x7f9ee8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:47.226 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.225+0000 7f9ef8ff9700 1 -- 172.21.15.174:0/1364927652 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9eec041000 con 0x7f9efc060280 2023-12-25T21:08:47.226 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.225+0000 7f9efb7fe700 1 --2- 172.21.15.174:0/1364927652 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f9ee802ff60 0x7f9ee8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:47.226 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.226+0000 7f9efb7fe700 1 --2- 172.21.15.174:0/1364927652 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f9ee802ff60 0x7f9ee8032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9ee40096a0 tx=0x7f9ee4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:47.229 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.230+0000 7f9ef8ff9700 1 -- 172.21.15.174:0/1364927652 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9eec014420 con 0x7f9efc060280 2023-12-25T21:08:47.437 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.437+0000 7f9f0267a700 1 -- 172.21.15.174:0/1364927652 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9efc108110 con 0x7f9efc060280 2023-12-25T21:08:47.438 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.438+0000 7f9ef8ff9700 1 -- 172.21.15.174:0/1364927652 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9eec010230 con 0x7f9efc060280 2023-12-25T21:08:47.438 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:47.438 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:47.440 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.440+0000 7f9f0267a700 1 -- 172.21.15.174:0/1364927652 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f9ee802ff60 msgr2=0x7f9ee8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:47.440 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.440+0000 7f9f0267a700 1 --2- 172.21.15.174:0/1364927652 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f9ee802ff60 0x7f9ee8032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9ee40096a0 tx=0x7f9ee4006b40).stop 2023-12-25T21:08:47.440 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.440+0000 7f9f0267a700 1 -- 172.21.15.174:0/1364927652 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9efc060280 msgr2=0x7f9efc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:47.440 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.440+0000 7f9f0267a700 1 --2- 172.21.15.174:0/1364927652 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9efc060280 0x7f9efc1066e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f9eec013510 tx=0x7f9eec00b040).stop 2023-12-25T21:08:47.440 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.440+0000 7f9f0267a700 1 -- 172.21.15.174:0/1364927652 shutdown_connections 2023-12-25T21:08:47.440 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.440+0000 7f9f0267a700 1 --2- 172.21.15.174:0/1364927652 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f9ee802ff60 0x7f9ee8032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:47.440 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.440+0000 7f9f0267a700 1 --2- 172.21.15.174:0/1364927652 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9efc060280 0x7f9efc1066e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:47.440 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.440+0000 7f9f0267a700 1 -- 172.21.15.174:0/1364927652 >> 172.21.15.174:0/1364927652 conn(0x7f9efc0f0b70 msgr2=0x7f9efc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:47.441 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.440+0000 7f9f0267a700 1 -- 172.21.15.174:0/1364927652 shutdown_connections 2023-12-25T21:08:47.441 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:47.440+0000 7f9f0267a700 1 -- 172.21.15.174:0/1364927652 wait complete. 2023-12-25T21:08:47.441 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:48.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:48 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:46.584022+0000 mgr.smithi061.jxxeke (mgr.14164) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:48.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:48 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:47.438511+0000 mon.smithi061 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.174:0/1364927652' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:48.851 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:48.851 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:50.005 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.004+0000 7f407d731700 1 -- 172.21.15.174:0/2847385339 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40780f4860 msgr2=0x7f40780f4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:50.005 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.004+0000 7f407d731700 1 --2- 172.21.15.174:0/2847385339 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40780f4860 0x7f40780f4c80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f406c009fb0 tx=0x7f406c009b30).stop 2023-12-25T21:08:50.005 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.004+0000 7f407d731700 1 -- 172.21.15.174:0/2847385339 shutdown_connections 2023-12-25T21:08:50.006 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.004+0000 7f407d731700 1 --2- 172.21.15.174:0/2847385339 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40780f4860 0x7f40780f4c80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:50.006 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.004+0000 7f407d731700 1 -- 172.21.15.174:0/2847385339 >> 172.21.15.174:0/2847385339 conn(0x7f40780f02e0 msgr2=0x7f40780f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:50.006 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.004+0000 7f407d731700 1 -- 172.21.15.174:0/2847385339 shutdown_connections 2023-12-25T21:08:50.006 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.004+0000 7f407d731700 1 -- 172.21.15.174:0/2847385339 wait complete. 2023-12-25T21:08:50.006 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.005+0000 7f407d731700 1 Processor -- start 2023-12-25T21:08:50.006 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.005+0000 7f407d731700 1 -- start start 2023-12-25T21:08:50.006 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.005+0000 7f407d731700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40780f4860 0x7f40780f7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:50.006 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.005+0000 7f407d731700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40780f58d0 con 0x7f40780f4860 2023-12-25T21:08:50.006 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.005+0000 7f4076ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40780f4860 0x7f40780f7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:50.006 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.005+0000 7f4076ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40780f4860 0x7f40780f7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:51852/0 (socket says 172.21.15.174:51852) 2023-12-25T21:08:50.006 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.005+0000 7f4076ffd700 1 -- 172.21.15.174:0/379783274 learned_addr learned my addr 172.21.15.174:0/379783274 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:50.006 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.006+0000 7f4076ffd700 1 -- 172.21.15.174:0/379783274 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c005040 con 0x7f40780f4860 2023-12-25T21:08:50.006 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.006+0000 7f4076ffd700 1 --2- 172.21.15.174:0/379783274 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40780f4860 0x7f40780f7450 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f406c009830 tx=0x7f406c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:50.008 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.006+0000 7f405f7fe700 1 -- 172.21.15.174:0/379783274 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f406c009320 con 0x7f40780f4860 2023-12-25T21:08:50.008 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.006+0000 7f407d731700 1 -- 172.21.15.174:0/379783274 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40780f5a70 con 0x7f40780f4860 2023-12-25T21:08:50.008 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.006+0000 7f405f7fe700 1 -- 172.21.15.174:0/379783274 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f406c0136c0 con 0x7f40780f4860 2023-12-25T21:08:50.008 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.006+0000 7f405f7fe700 1 -- 172.21.15.174:0/379783274 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f406c01ee90 con 0x7f40780f4860 2023-12-25T21:08:50.008 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.006+0000 7f407d731700 1 -- 172.21.15.174:0/379783274 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40780f5f00 con 0x7f40780f4860 2023-12-25T21:08:50.008 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.007+0000 7f405f7fe700 1 -- 172.21.15.174:0/379783274 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f406c01e420 con 0x7f40780f4860 2023-12-25T21:08:50.008 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.007+0000 7f407d731700 1 -- 172.21.15.174:0/379783274 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4078059100 con 0x7f40780f4860 2023-12-25T21:08:50.008 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.007+0000 7f405f7fe700 1 --2- 172.21.15.174:0/379783274 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f406002ff10 0x7f40600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:50.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.007+0000 7f405f7fe700 1 -- 172.21.15.174:0/379783274 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f406c041020 con 0x7f40780f4860 2023-12-25T21:08:50.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.007+0000 7f40767fc700 1 --2- 172.21.15.174:0/379783274 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f406002ff10 0x7f40600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:50.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.008+0000 7f40767fc700 1 --2- 172.21.15.174:0/379783274 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f406002ff10 0x7f40600323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f40680096a0 tx=0x7f4068006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:50.014 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.012+0000 7f405f7fe700 1 -- 172.21.15.174:0/379783274 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f406c0095f0 con 0x7f40780f4860 2023-12-25T21:08:50.218 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.218+0000 7f407d731700 1 -- 172.21.15.174:0/379783274 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4078045ad0 con 0x7f40780f4860 2023-12-25T21:08:50.219 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.219+0000 7f405f7fe700 1 -- 172.21.15.174:0/379783274 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f406c02c2d0 con 0x7f40780f4860 2023-12-25T21:08:50.219 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:50.219 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:50.221 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.221+0000 7f407d731700 1 -- 172.21.15.174:0/379783274 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f406002ff10 msgr2=0x7f40600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:50.221 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.221+0000 7f407d731700 1 --2- 172.21.15.174:0/379783274 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f406002ff10 0x7f40600323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f40680096a0 tx=0x7f4068006b40).stop 2023-12-25T21:08:50.221 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.221+0000 7f407d731700 1 -- 172.21.15.174:0/379783274 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40780f4860 msgr2=0x7f40780f7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:50.221 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.221+0000 7f407d731700 1 --2- 172.21.15.174:0/379783274 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40780f4860 0x7f40780f7450 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f406c009830 tx=0x7f406c00b040).stop 2023-12-25T21:08:50.221 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.221+0000 7f407d731700 1 -- 172.21.15.174:0/379783274 shutdown_connections 2023-12-25T21:08:50.221 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.221+0000 7f407d731700 1 --2- 172.21.15.174:0/379783274 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f406002ff10 0x7f40600323d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:50.221 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.221+0000 7f407d731700 1 --2- 172.21.15.174:0/379783274 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40780f4860 0x7f40780f7450 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:50.221 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.221+0000 7f407d731700 1 -- 172.21.15.174:0/379783274 >> 172.21.15.174:0/379783274 conn(0x7f40780f02e0 msgr2=0x7f40780f0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:50.221 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.221+0000 7f407d731700 1 -- 172.21.15.174:0/379783274 shutdown_connections 2023-12-25T21:08:50.222 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:50.221+0000 7f407d731700 1 -- 172.21.15.174:0/379783274 wait complete. 2023-12-25T21:08:50.222 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:50.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:50 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:48.584430+0000 mgr.smithi061.jxxeke (mgr.14164) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:50.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:50 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:50.219536+0000 mon.smithi061 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.174:0/379783274' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:51.612 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:51.613 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:52.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:52 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:50.584800+0000 mgr.smithi061.jxxeke (mgr.14164) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:52.756 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.754+0000 7ff6640de700 1 -- 172.21.15.174:0/1668038808 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff65c0f5410 msgr2=0x7ff65c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:52.756 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.754+0000 7ff6640de700 1 --2- 172.21.15.174:0/1668038808 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff65c0f5410 0x7ff65c0f5830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7ff64c009fb0 tx=0x7ff64c009b30).stop 2023-12-25T21:08:52.756 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.754+0000 7ff6640de700 1 -- 172.21.15.174:0/1668038808 shutdown_connections 2023-12-25T21:08:52.756 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.754+0000 7ff6640de700 1 --2- 172.21.15.174:0/1668038808 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff65c0f5410 0x7ff65c0f5830 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:52.756 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.754+0000 7ff6640de700 1 -- 172.21.15.174:0/1668038808 >> 172.21.15.174:0/1668038808 conn(0x7ff65c0f0b70 msgr2=0x7ff65c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:52.756 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.754+0000 7ff6640de700 1 -- 172.21.15.174:0/1668038808 shutdown_connections 2023-12-25T21:08:52.756 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.754+0000 7ff6640de700 1 -- 172.21.15.174:0/1668038808 wait complete. 2023-12-25T21:08:52.756 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.755+0000 7ff6640de700 1 Processor -- start 2023-12-25T21:08:52.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.755+0000 7ff6640de700 1 -- start start 2023-12-25T21:08:52.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.755+0000 7ff6640de700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff65c0f5410 0x7ff65c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:52.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.755+0000 7ff6640de700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c105b20 con 0x7ff65c0f5410 2023-12-25T21:08:52.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.755+0000 7ff661e7a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff65c0f5410 0x7ff65c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:52.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.756+0000 7ff661e7a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff65c0f5410 0x7ff65c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:51884/0 (socket says 172.21.15.174:51884) 2023-12-25T21:08:52.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.756+0000 7ff661e7a700 1 -- 172.21.15.174:0/2999188442 learned_addr learned my addr 172.21.15.174:0/2999188442 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:52.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.756+0000 7ff661e7a700 1 -- 172.21.15.174:0/2999188442 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64c005040 con 0x7ff65c0f5410 2023-12-25T21:08:52.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.756+0000 7ff661e7a700 1 --2- 172.21.15.174:0/2999188442 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff65c0f5410 0x7ff65c105630 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7ff64c013510 tx=0x7ff64c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:52.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.756+0000 7ff6527fc700 1 -- 172.21.15.174:0/2999188442 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff64c009320 con 0x7ff65c0f5410 2023-12-25T21:08:52.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.756+0000 7ff6640de700 1 -- 172.21.15.174:0/2999188442 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff65c105d20 con 0x7ff65c0f5410 2023-12-25T21:08:52.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.756+0000 7ff6527fc700 1 -- 172.21.15.174:0/2999188442 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff64c013f00 con 0x7ff65c0f5410 2023-12-25T21:08:52.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.756+0000 7ff6527fc700 1 -- 172.21.15.174:0/2999188442 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff64c014d90 con 0x7ff65c0f5410 2023-12-25T21:08:52.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.756+0000 7ff6640de700 1 -- 172.21.15.174:0/2999188442 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff65c1061b0 con 0x7ff65c0f5410 2023-12-25T21:08:52.758 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.757+0000 7ff6527fc700 1 -- 172.21.15.174:0/2999188442 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff64c0095f0 con 0x7ff65c0f5410 2023-12-25T21:08:52.758 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.757+0000 7ff6640de700 1 -- 172.21.15.174:0/2999188442 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff65c045ad0 con 0x7ff65c0f5410 2023-12-25T21:08:52.758 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.757+0000 7ff6527fc700 1 --2- 172.21.15.174:0/2999188442 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff64802ff60 0x7ff648032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:52.758 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.757+0000 7ff6527fc700 1 -- 172.21.15.174:0/2999188442 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff64c0410c0 con 0x7ff65c0f5410 2023-12-25T21:08:52.758 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.757+0000 7ff661679700 1 --2- 172.21.15.174:0/2999188442 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff64802ff60 0x7ff648032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:52.759 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.758+0000 7ff661679700 1 --2- 172.21.15.174:0/2999188442 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff64802ff60 0x7ff648032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7ff6580096a0 tx=0x7ff658006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:52.761 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.762+0000 7ff6527fc700 1 -- 172.21.15.174:0/2999188442 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff64c014420 con 0x7ff65c0f5410 2023-12-25T21:08:52.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.966+0000 7ff6640de700 1 -- 172.21.15.174:0/2999188442 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff65c0fae60 con 0x7ff65c0f5410 2023-12-25T21:08:52.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.967+0000 7ff6527fc700 1 -- 172.21.15.174:0/2999188442 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff64c01d7f0 con 0x7ff65c0f5410 2023-12-25T21:08:52.967 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:52.967 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:52.968 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.968+0000 7ff6640de700 1 -- 172.21.15.174:0/2999188442 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff64802ff60 msgr2=0x7ff648032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:52.968 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.968+0000 7ff6640de700 1 --2- 172.21.15.174:0/2999188442 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff64802ff60 0x7ff648032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7ff6580096a0 tx=0x7ff658006b40).stop 2023-12-25T21:08:52.968 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.968+0000 7ff6640de700 1 -- 172.21.15.174:0/2999188442 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff65c0f5410 msgr2=0x7ff65c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:52.968 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.968+0000 7ff6640de700 1 --2- 172.21.15.174:0/2999188442 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff65c0f5410 0x7ff65c105630 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7ff64c013510 tx=0x7ff64c00b040).stop 2023-12-25T21:08:52.969 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.968+0000 7ff6640de700 1 -- 172.21.15.174:0/2999188442 shutdown_connections 2023-12-25T21:08:52.969 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.968+0000 7ff6640de700 1 --2- 172.21.15.174:0/2999188442 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff64802ff60 0x7ff648032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:52.969 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.968+0000 7ff6640de700 1 --2- 172.21.15.174:0/2999188442 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff65c0f5410 0x7ff65c105630 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:52.969 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.968+0000 7ff6640de700 1 -- 172.21.15.174:0/2999188442 >> 172.21.15.174:0/2999188442 conn(0x7ff65c0f0b70 msgr2=0x7ff65c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:52.969 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.968+0000 7ff6640de700 1 -- 172.21.15.174:0/2999188442 shutdown_connections 2023-12-25T21:08:52.969 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:52.968+0000 7ff6640de700 1 -- 172.21.15.174:0/2999188442 wait complete. 2023-12-25T21:08:52.969 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:53.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:52.702716+0000 mon.smithi061 (mon.0) 152 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/mirror_snapshot_schedule"}]: dispatch 2023-12-25T21:08:53.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:52.704659+0000 mon.smithi061 (mon.0) 153 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/trash_purge_schedule"}]: dispatch 2023-12-25T21:08:53.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:52.967168+0000 mon.smithi061 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.174:0/2999188442' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:54.352 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:54.352 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:54.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:54 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:52.585239+0000 mgr.smithi061.jxxeke (mgr.14164) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:55.564 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.563+0000 7ff1ba44d700 1 -- 172.21.15.174:0/1016627519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b40f51a0 msgr2=0x7ff1b40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:55.564 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.563+0000 7ff1ba44d700 1 --2- 172.21.15.174:0/1016627519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b40f51a0 0x7ff1b40f55c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7ff1b0009fb0 tx=0x7ff1b0009b30).stop 2023-12-25T21:08:55.564 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.563+0000 7ff1ba44d700 1 -- 172.21.15.174:0/1016627519 shutdown_connections 2023-12-25T21:08:55.565 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.563+0000 7ff1ba44d700 1 --2- 172.21.15.174:0/1016627519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b40f51a0 0x7ff1b40f55c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:55.565 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.563+0000 7ff1ba44d700 1 -- 172.21.15.174:0/1016627519 >> 172.21.15.174:0/1016627519 conn(0x7ff1b40f09a0 msgr2=0x7ff1b40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:55.565 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.563+0000 7ff1ba44d700 1 -- 172.21.15.174:0/1016627519 shutdown_connections 2023-12-25T21:08:55.565 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.563+0000 7ff1ba44d700 1 -- 172.21.15.174:0/1016627519 wait complete. 2023-12-25T21:08:55.565 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.564+0000 7ff1ba44d700 1 Processor -- start 2023-12-25T21:08:55.565 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.564+0000 7ff1ba44d700 1 -- start start 2023-12-25T21:08:55.565 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.564+0000 7ff1ba44d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b40f51a0 0x7ff1b410b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:55.565 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.564+0000 7ff1ba44d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b4106fc0 con 0x7ff1b40f51a0 2023-12-25T21:08:55.565 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.564+0000 7ff1b944b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b40f51a0 0x7ff1b410b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:55.565 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.564+0000 7ff1b944b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b40f51a0 0x7ff1b410b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:51896/0 (socket says 172.21.15.174:51896) 2023-12-25T21:08:55.565 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.564+0000 7ff1b944b700 1 -- 172.21.15.174:0/3152200532 learned_addr learned my addr 172.21.15.174:0/3152200532 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:55.565 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.564+0000 7ff1b944b700 1 -- 172.21.15.174:0/3152200532 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b0005040 con 0x7ff1b40f51a0 2023-12-25T21:08:55.565 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.564+0000 7ff1b944b700 1 --2- 172.21.15.174:0/3152200532 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b40f51a0 0x7ff1b410b5d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7ff1b0013510 tx=0x7ff1b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:55.566 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.565+0000 7ff1a9ffb700 1 -- 172.21.15.174:0/3152200532 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1b0009320 con 0x7ff1b40f51a0 2023-12-25T21:08:55.566 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.565+0000 7ff1ba44d700 1 -- 172.21.15.174:0/3152200532 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b4060220 con 0x7ff1b40f51a0 2023-12-25T21:08:55.566 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.565+0000 7ff1a9ffb700 1 -- 172.21.15.174:0/3152200532 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff1b0013f00 con 0x7ff1b40f51a0 2023-12-25T21:08:55.566 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.565+0000 7ff1a9ffb700 1 -- 172.21.15.174:0/3152200532 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1b0014dc0 con 0x7ff1b40f51a0 2023-12-25T21:08:55.566 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.565+0000 7ff1ba44d700 1 -- 172.21.15.174:0/3152200532 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1b410bac0 con 0x7ff1b40f51a0 2023-12-25T21:08:55.569 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.566+0000 7ff1a9ffb700 1 -- 172.21.15.174:0/3152200532 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff1b001d3d0 con 0x7ff1b40f51a0 2023-12-25T21:08:55.569 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.566+0000 7ff1a9ffb700 1 --2- 172.21.15.174:0/3152200532 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff1a002ff60 0x7ff1a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:55.569 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.566+0000 7ff1ba44d700 1 -- 172.21.15.174:0/3152200532 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1b4045ad0 con 0x7ff1b40f51a0 2023-12-25T21:08:55.569 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.566+0000 7ff1a9ffb700 1 -- 172.21.15.174:0/3152200532 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff1b0041010 con 0x7ff1b40f51a0 2023-12-25T21:08:55.569 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.566+0000 7ff1b8c4a700 1 --2- 172.21.15.174:0/3152200532 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff1a002ff60 0x7ff1a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:55.569 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.566+0000 7ff1b8c4a700 1 --2- 172.21.15.174:0/3152200532 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff1a002ff60 0x7ff1a0032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7ff1a40096a0 tx=0x7ff1a4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:55.570 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.571+0000 7ff1a9ffb700 1 -- 172.21.15.174:0/3152200532 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff1b006d2b0 con 0x7ff1b40f51a0 2023-12-25T21:08:55.774 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.774+0000 7ff1ba44d700 1 -- 172.21.15.174:0/3152200532 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1b410c2b0 con 0x7ff1b40f51a0 2023-12-25T21:08:55.774 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.775+0000 7ff1a9ffb700 1 -- 172.21.15.174:0/3152200532 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff1b0012510 con 0x7ff1b40f51a0 2023-12-25T21:08:55.776 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:55.776 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:55.777 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.776+0000 7ff1ba44d700 1 -- 172.21.15.174:0/3152200532 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff1a002ff60 msgr2=0x7ff1a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:55.777 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.776+0000 7ff1ba44d700 1 --2- 172.21.15.174:0/3152200532 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff1a002ff60 0x7ff1a0032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7ff1a40096a0 tx=0x7ff1a4006b40).stop 2023-12-25T21:08:55.778 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.776+0000 7ff1ba44d700 1 -- 172.21.15.174:0/3152200532 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b40f51a0 msgr2=0x7ff1b410b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:55.778 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.776+0000 7ff1ba44d700 1 --2- 172.21.15.174:0/3152200532 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b40f51a0 0x7ff1b410b5d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7ff1b0013510 tx=0x7ff1b000b040).stop 2023-12-25T21:08:55.778 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.776+0000 7ff1ba44d700 1 -- 172.21.15.174:0/3152200532 shutdown_connections 2023-12-25T21:08:55.778 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.776+0000 7ff1ba44d700 1 --2- 172.21.15.174:0/3152200532 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff1a002ff60 0x7ff1a0032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:55.778 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.776+0000 7ff1ba44d700 1 --2- 172.21.15.174:0/3152200532 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1b40f51a0 0x7ff1b410b5d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:55.778 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.776+0000 7ff1ba44d700 1 -- 172.21.15.174:0/3152200532 >> 172.21.15.174:0/3152200532 conn(0x7ff1b40f09a0 msgr2=0x7ff1b40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:55.778 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.777+0000 7ff1ba44d700 1 -- 172.21.15.174:0/3152200532 shutdown_connections 2023-12-25T21:08:55.778 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:55.777+0000 7ff1ba44d700 1 -- 172.21.15.174:0/3152200532 wait complete. 2023-12-25T21:08:55.778 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:56.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:56 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:54.585631+0000 mgr.smithi061.jxxeke (mgr.14164) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:56.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:56 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:55.775347+0000 mon.smithi061 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.174:0/3152200532' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:57.171 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:57.171 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:08:58.305 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.304+0000 7f7d7f59e700 1 -- 172.21.15.174:0/1222646101 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800f57b0 msgr2=0x7f7d800f5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:58.305 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.304+0000 7f7d7f59e700 1 --2- 172.21.15.174:0/1222646101 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800f57b0 0x7f7d800f5bd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f7d74009fb0 tx=0x7f7d74009b30).stop 2023-12-25T21:08:58.305 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.304+0000 7f7d7f59e700 1 -- 172.21.15.174:0/1222646101 shutdown_connections 2023-12-25T21:08:58.305 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.304+0000 7f7d7f59e700 1 --2- 172.21.15.174:0/1222646101 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800f57b0 0x7f7d800f5bd0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:58.305 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.304+0000 7f7d7f59e700 1 -- 172.21.15.174:0/1222646101 >> 172.21.15.174:0/1222646101 conn(0x7f7d800f0120 msgr2=0x7f7d800f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:58.306 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.304+0000 7f7d7f59e700 1 -- 172.21.15.174:0/1222646101 shutdown_connections 2023-12-25T21:08:58.306 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.304+0000 7f7d7f59e700 1 -- 172.21.15.174:0/1222646101 wait complete. 2023-12-25T21:08:58.306 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.304+0000 7f7d7f59e700 1 Processor -- start 2023-12-25T21:08:58.306 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.304+0000 7f7d7f59e700 1 -- start start 2023-12-25T21:08:58.306 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.304+0000 7f7d7f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800f57b0 0x7f7d801053c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:58.306 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.304+0000 7f7d7f59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d801058b0 con 0x7f7d800f57b0 2023-12-25T21:08:58.306 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.305+0000 7f7d7e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800f57b0 0x7f7d801053c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:58.306 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.305+0000 7f7d7e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800f57b0 0x7f7d801053c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:38702/0 (socket says 172.21.15.174:38702) 2023-12-25T21:08:58.306 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.305+0000 7f7d7e59c700 1 -- 172.21.15.174:0/1166028229 learned_addr learned my addr 172.21.15.174:0/1166028229 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:08:58.306 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.305+0000 7f7d7e59c700 1 -- 172.21.15.174:0/1166028229 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d74005040 con 0x7f7d800f57b0 2023-12-25T21:08:58.307 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.305+0000 7f7d7e59c700 1 --2- 172.21.15.174:0/1166028229 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800f57b0 0x7f7d801053c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f7d74013510 tx=0x7f7d7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:58.307 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.305+0000 7f7d6effd700 1 -- 172.21.15.174:0/1166028229 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d74009320 con 0x7f7d800f57b0 2023-12-25T21:08:58.307 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.305+0000 7f7d7f59e700 1 -- 172.21.15.174:0/1166028229 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d80105ab0 con 0x7f7d800f57b0 2023-12-25T21:08:58.307 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.305+0000 7f7d6effd700 1 -- 172.21.15.174:0/1166028229 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d74013f00 con 0x7f7d800f57b0 2023-12-25T21:08:58.307 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.305+0000 7f7d6effd700 1 -- 172.21.15.174:0/1166028229 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d74014dc0 con 0x7f7d800f57b0 2023-12-25T21:08:58.307 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.305+0000 7f7d7f59e700 1 -- 172.21.15.174:0/1166028229 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d80105ec0 con 0x7f7d800f57b0 2023-12-25T21:08:58.308 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.306+0000 7f7d6effd700 1 -- 172.21.15.174:0/1166028229 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7d74014420 con 0x7f7d800f57b0 2023-12-25T21:08:58.309 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.306+0000 7f7d7f59e700 1 -- 172.21.15.174:0/1166028229 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d80045ad0 con 0x7f7d800f57b0 2023-12-25T21:08:58.309 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.306+0000 7f7d6effd700 1 --2- 172.21.15.174:0/1166028229 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7d68032170 0x7f7d68034630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:08:58.309 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.306+0000 7f7d6effd700 1 -- 172.21.15.174:0/1166028229 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7d740410d0 con 0x7f7d800f57b0 2023-12-25T21:08:58.310 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.306+0000 7f7d7dd9b700 1 --2- 172.21.15.174:0/1166028229 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7d68032170 0x7f7d68034630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:08:58.310 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.307+0000 7f7d7dd9b700 1 --2- 172.21.15.174:0/1166028229 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7d68032170 0x7f7d68034630 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f7d700096a0 tx=0x7f7d70006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:08:58.311 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.311+0000 7f7d6effd700 1 -- 172.21.15.174:0/1166028229 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d740095f0 con 0x7f7d800f57b0 2023-12-25T21:08:58.518 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.518+0000 7f7d7f59e700 1 -- 172.21.15.174:0/1166028229 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d80106d80 con 0x7f7d800f57b0 2023-12-25T21:08:58.518 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.519+0000 7f7d6effd700 1 -- 172.21.15.174:0/1166028229 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7d7401cb10 con 0x7f7d800f57b0 2023-12-25T21:08:58.519 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:08:58.519 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:08:58.520 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.520+0000 7f7d7f59e700 1 -- 172.21.15.174:0/1166028229 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7d68032170 msgr2=0x7f7d68034630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:58.520 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.520+0000 7f7d7f59e700 1 --2- 172.21.15.174:0/1166028229 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7d68032170 0x7f7d68034630 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f7d700096a0 tx=0x7f7d70006b40).stop 2023-12-25T21:08:58.520 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.520+0000 7f7d7f59e700 1 -- 172.21.15.174:0/1166028229 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800f57b0 msgr2=0x7f7d801053c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:08:58.520 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.520+0000 7f7d7f59e700 1 --2- 172.21.15.174:0/1166028229 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800f57b0 0x7f7d801053c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f7d74013510 tx=0x7f7d7400b040).stop 2023-12-25T21:08:58.521 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.520+0000 7f7d7f59e700 1 -- 172.21.15.174:0/1166028229 shutdown_connections 2023-12-25T21:08:58.521 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.520+0000 7f7d7f59e700 1 --2- 172.21.15.174:0/1166028229 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7d68032170 0x7f7d68034630 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:58.521 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.520+0000 7f7d7f59e700 1 --2- 172.21.15.174:0/1166028229 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800f57b0 0x7f7d801053c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:08:58.521 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.520+0000 7f7d7f59e700 1 -- 172.21.15.174:0/1166028229 >> 172.21.15.174:0/1166028229 conn(0x7f7d800f0120 msgr2=0x7f7d800f0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:08:58.521 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.520+0000 7f7d7f59e700 1 -- 172.21.15.174:0/1166028229 shutdown_connections 2023-12-25T21:08:58.521 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:08:58.520+0000 7f7d7f59e700 1 -- 172.21.15.174:0/1166028229 wait complete. 2023-12-25T21:08:58.521 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:08:58.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:58 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:56.585966+0000 mgr.smithi061.jxxeke (mgr.14164) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:08:59.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:08:59 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:08:58.519176+0000 mon.smithi061 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.174:0/1166028229' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:08:59.931 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:08:59.931 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:00.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:00 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:08:58.586263+0000 mgr.smithi061.jxxeke (mgr.14164) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:01.063 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.062+0000 7f3dec5c2700 1 -- 172.21.15.174:0/485601772 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3de40f5410 msgr2=0x7f3de40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:01.064 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.062+0000 7f3dec5c2700 1 --2- 172.21.15.174:0/485601772 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3de40f5410 0x7f3de40f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f3dd4009fb0 tx=0x7f3dd4009b30).stop 2023-12-25T21:09:01.064 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.062+0000 7f3dec5c2700 1 -- 172.21.15.174:0/485601772 shutdown_connections 2023-12-25T21:09:01.064 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.062+0000 7f3dec5c2700 1 --2- 172.21.15.174:0/485601772 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3de40f5410 0x7f3de40f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:01.064 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.062+0000 7f3dec5c2700 1 -- 172.21.15.174:0/485601772 >> 172.21.15.174:0/485601772 conn(0x7f3de40f0b70 msgr2=0x7f3de40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:01.064 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.062+0000 7f3dec5c2700 1 -- 172.21.15.174:0/485601772 shutdown_connections 2023-12-25T21:09:01.064 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.062+0000 7f3dec5c2700 1 -- 172.21.15.174:0/485601772 wait complete. 2023-12-25T21:09:01.064 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.063+0000 7f3dec5c2700 1 Processor -- start 2023-12-25T21:09:01.064 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.063+0000 7f3dec5c2700 1 -- start start 2023-12-25T21:09:01.064 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.063+0000 7f3dec5c2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3de40f5410 0x7f3de410b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:01.064 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.063+0000 7f3dec5c2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de4107240 con 0x7f3de40f5410 2023-12-25T21:09:01.064 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.063+0000 7f3dea35e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3de40f5410 0x7f3de410b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:01.065 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.063+0000 7f3dea35e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3de40f5410 0x7f3de410b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:38718/0 (socket says 172.21.15.174:38718) 2023-12-25T21:09:01.065 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.063+0000 7f3dea35e700 1 -- 172.21.15.174:0/2816879170 learned_addr learned my addr 172.21.15.174:0/2816879170 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:01.065 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.064+0000 7f3dea35e700 1 -- 172.21.15.174:0/2816879170 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd4005040 con 0x7f3de40f5410 2023-12-25T21:09:01.065 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.064+0000 7f3dea35e700 1 --2- 172.21.15.174:0/2816879170 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3de40f5410 0x7f3de410b850 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f3dd4013510 tx=0x7f3dd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:01.065 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.064+0000 7f3ddaffd700 1 -- 172.21.15.174:0/2816879170 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dd4009320 con 0x7f3de40f5410 2023-12-25T21:09:01.066 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.064+0000 7f3dec5c2700 1 -- 172.21.15.174:0/2816879170 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de4060220 con 0x7f3de40f5410 2023-12-25T21:09:01.066 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.064+0000 7f3ddaffd700 1 -- 172.21.15.174:0/2816879170 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3dd4013f00 con 0x7f3de40f5410 2023-12-25T21:09:01.066 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.064+0000 7f3ddaffd700 1 -- 172.21.15.174:0/2816879170 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dd4014d90 con 0x7f3de40f5410 2023-12-25T21:09:01.066 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.064+0000 7f3dec5c2700 1 -- 172.21.15.174:0/2816879170 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de410bd40 con 0x7f3de40f5410 2023-12-25T21:09:01.066 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.065+0000 7f3ddaffd700 1 -- 172.21.15.174:0/2816879170 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3dd40095f0 con 0x7f3de40f5410 2023-12-25T21:09:01.066 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.065+0000 7f3dec5c2700 1 -- 172.21.15.174:0/2816879170 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de4045ad0 con 0x7f3de40f5410 2023-12-25T21:09:01.066 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.065+0000 7f3ddaffd700 1 --2- 172.21.15.174:0/2816879170 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3dd002ff60 0x7f3dd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:01.066 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.065+0000 7f3ddaffd700 1 -- 172.21.15.174:0/2816879170 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3dd4040fe0 con 0x7f3de40f5410 2023-12-25T21:09:01.066 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.065+0000 7f3de9b5d700 1 --2- 172.21.15.174:0/2816879170 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3dd002ff60 0x7f3dd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:01.066 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.065+0000 7f3de9b5d700 1 --2- 172.21.15.174:0/2816879170 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3dd002ff60 0x7f3dd0032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f3de00096a0 tx=0x7f3de0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:01.069 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.070+0000 7f3ddaffd700 1 -- 172.21.15.174:0/2816879170 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3dd401d7f0 con 0x7f3de40f5410 2023-12-25T21:09:01.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.275+0000 7f3dec5c2700 1 -- 172.21.15.174:0/2816879170 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3de40fae60 con 0x7f3de40f5410 2023-12-25T21:09:01.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.276+0000 7f3ddaffd700 1 -- 172.21.15.174:0/2816879170 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3dd4010230 con 0x7f3de40f5410 2023-12-25T21:09:01.275 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:01.276 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:01.277 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.277+0000 7f3dec5c2700 1 -- 172.21.15.174:0/2816879170 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3dd002ff60 msgr2=0x7f3dd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:01.277 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.277+0000 7f3dec5c2700 1 --2- 172.21.15.174:0/2816879170 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3dd002ff60 0x7f3dd0032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f3de00096a0 tx=0x7f3de0006b40).stop 2023-12-25T21:09:01.277 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.277+0000 7f3dec5c2700 1 -- 172.21.15.174:0/2816879170 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3de40f5410 msgr2=0x7f3de410b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:01.277 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.277+0000 7f3dec5c2700 1 --2- 172.21.15.174:0/2816879170 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3de40f5410 0x7f3de410b850 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f3dd4013510 tx=0x7f3dd400b040).stop 2023-12-25T21:09:01.277 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.277+0000 7f3dec5c2700 1 -- 172.21.15.174:0/2816879170 shutdown_connections 2023-12-25T21:09:01.277 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.277+0000 7f3dec5c2700 1 --2- 172.21.15.174:0/2816879170 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3dd002ff60 0x7f3dd0032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:01.278 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.277+0000 7f3dec5c2700 1 --2- 172.21.15.174:0/2816879170 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3de40f5410 0x7f3de410b850 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:01.278 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.277+0000 7f3dec5c2700 1 -- 172.21.15.174:0/2816879170 >> 172.21.15.174:0/2816879170 conn(0x7f3de40f0b70 msgr2=0x7f3de40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:01.278 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.277+0000 7f3dec5c2700 1 -- 172.21.15.174:0/2816879170 shutdown_connections 2023-12-25T21:09:01.278 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:01.277+0000 7f3dec5c2700 1 -- 172.21.15.174:0/2816879170 wait complete. 2023-12-25T21:09:01.278 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:01.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:01.275943+0000 mon.smithi061 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.174:0/2816879170' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:02.672 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:02.673 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:02.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:02 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:00.586580+0000 mgr.smithi061.jxxeke (mgr.14164) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:03.869 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.868+0000 7fb169ade700 1 -- 172.21.15.174:0/55802123 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb164060280 msgr2=0x7fb1640f90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:03.869 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.868+0000 7fb169ade700 1 --2- 172.21.15.174:0/55802123 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb164060280 0x7fb1640f90d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fb14c009fb0 tx=0x7fb14c009b30).stop 2023-12-25T21:09:03.869 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.868+0000 7fb169ade700 1 -- 172.21.15.174:0/55802123 shutdown_connections 2023-12-25T21:09:03.870 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.868+0000 7fb169ade700 1 --2- 172.21.15.174:0/55802123 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb164060280 0x7fb1640f90d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:03.870 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.868+0000 7fb169ade700 1 -- 172.21.15.174:0/55802123 >> 172.21.15.174:0/55802123 conn(0x7fb1640f0b70 msgr2=0x7fb1640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:03.870 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.868+0000 7fb169ade700 1 -- 172.21.15.174:0/55802123 shutdown_connections 2023-12-25T21:09:03.870 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.868+0000 7fb169ade700 1 -- 172.21.15.174:0/55802123 wait complete. 2023-12-25T21:09:03.871 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.869+0000 7fb169ade700 1 Processor -- start 2023-12-25T21:09:03.871 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.869+0000 7fb169ade700 1 -- start start 2023-12-25T21:09:03.871 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.870+0000 7fb169ade700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb164060280 0x7fb1640f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:03.871 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.870+0000 7fb169ade700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1640f4fd0 con 0x7fb164060280 2023-12-25T21:09:03.871 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.870+0000 7fb1637fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb164060280 0x7fb1640f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:03.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.870+0000 7fb1637fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb164060280 0x7fb1640f6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:38746/0 (socket says 172.21.15.174:38746) 2023-12-25T21:09:03.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.870+0000 7fb1637fe700 1 -- 172.21.15.174:0/483753918 learned_addr learned my addr 172.21.15.174:0/483753918 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:03.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.870+0000 7fb1637fe700 1 -- 172.21.15.174:0/483753918 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb14c005040 con 0x7fb164060280 2023-12-25T21:09:03.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.870+0000 7fb1637fe700 1 --2- 172.21.15.174:0/483753918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb164060280 0x7fb1640f6b50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fb14c009830 tx=0x7fb14c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:03.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.870+0000 7fb15bfff700 1 -- 172.21.15.174:0/483753918 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb14c009320 con 0x7fb164060280 2023-12-25T21:09:03.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.870+0000 7fb169ade700 1 -- 172.21.15.174:0/483753918 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1640f5170 con 0x7fb164060280 2023-12-25T21:09:03.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.870+0000 7fb15bfff700 1 -- 172.21.15.174:0/483753918 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb14c0136c0 con 0x7fb164060280 2023-12-25T21:09:03.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.870+0000 7fb15bfff700 1 -- 172.21.15.174:0/483753918 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb14c01ee90 con 0x7fb164060280 2023-12-25T21:09:03.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.871+0000 7fb169ade700 1 -- 172.21.15.174:0/483753918 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1640f5600 con 0x7fb164060280 2023-12-25T21:09:03.873 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.871+0000 7fb15bfff700 1 -- 172.21.15.174:0/483753918 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb14c01e420 con 0x7fb164060280 2023-12-25T21:09:03.873 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.871+0000 7fb169ade700 1 -- 172.21.15.174:0/483753918 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb164059100 con 0x7fb164060280 2023-12-25T21:09:03.873 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.872+0000 7fb15bfff700 1 --2- 172.21.15.174:0/483753918 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb15002ff10 0x7fb1500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:03.873 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.872+0000 7fb15bfff700 1 -- 172.21.15.174:0/483753918 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb14c040f30 con 0x7fb164060280 2023-12-25T21:09:03.873 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.872+0000 7fb162ffd700 1 --2- 172.21.15.174:0/483753918 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb15002ff10 0x7fb1500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:03.873 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.872+0000 7fb162ffd700 1 --2- 172.21.15.174:0/483753918 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb15002ff10 0x7fb1500323d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fb1540096a0 tx=0x7fb154006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:03.877 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:03.876+0000 7fb15bfff700 1 -- 172.21.15.174:0/483753918 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb14c06d2b0 con 0x7fb164060280 2023-12-25T21:09:04.080 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:04.079+0000 7fb169ade700 1 -- 172.21.15.174:0/483753918 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1640fbf30 con 0x7fb164060280 2023-12-25T21:09:04.081 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:04.080+0000 7fb15bfff700 1 -- 172.21.15.174:0/483753918 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb14c01c790 con 0x7fb164060280 2023-12-25T21:09:04.081 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:04.081 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:04.082 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:04.081+0000 7fb169ade700 1 -- 172.21.15.174:0/483753918 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb15002ff10 msgr2=0x7fb1500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:04.082 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:04.081+0000 7fb169ade700 1 --2- 172.21.15.174:0/483753918 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb15002ff10 0x7fb1500323d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fb1540096a0 tx=0x7fb154006b40).stop 2023-12-25T21:09:04.082 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:04.081+0000 7fb169ade700 1 -- 172.21.15.174:0/483753918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb164060280 msgr2=0x7fb1640f6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:04.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:04.082+0000 7fb169ade700 1 --2- 172.21.15.174:0/483753918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb164060280 0x7fb1640f6b50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fb14c009830 tx=0x7fb14c00b040).stop 2023-12-25T21:09:04.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:04.082+0000 7fb169ade700 1 -- 172.21.15.174:0/483753918 shutdown_connections 2023-12-25T21:09:04.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:04.082+0000 7fb169ade700 1 --2- 172.21.15.174:0/483753918 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb15002ff10 0x7fb1500323d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:04.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:04.082+0000 7fb169ade700 1 --2- 172.21.15.174:0/483753918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb164060280 0x7fb1640f6b50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:04.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:04.082+0000 7fb169ade700 1 -- 172.21.15.174:0/483753918 >> 172.21.15.174:0/483753918 conn(0x7fb1640f0b70 msgr2=0x7fb1640f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:04.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:04.082+0000 7fb169ade700 1 -- 172.21.15.174:0/483753918 shutdown_connections 2023-12-25T21:09:04.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:04.082+0000 7fb169ade700 1 -- 172.21.15.174:0/483753918 wait complete. 2023-12-25T21:09:04.083 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:04.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:04 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:02.586835+0000 mgr.smithi061.jxxeke (mgr.14164) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:04.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:04 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:04.081388+0000 mon.smithi061 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.174:0/483753918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:05.468 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:05.469 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:06.650 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.647+0000 7f89757f9700 1 -- 172.21.15.174:0/1540119130 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89700f5410 msgr2=0x7f89700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:06.650 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.647+0000 7f89757f9700 1 --2- 172.21.15.174:0/1540119130 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89700f5410 0x7f89700f5830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f8964009fb0 tx=0x7f8964009b30).stop 2023-12-25T21:09:06.650 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.647+0000 7f89757f9700 1 -- 172.21.15.174:0/1540119130 shutdown_connections 2023-12-25T21:09:06.650 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.647+0000 7f89757f9700 1 --2- 172.21.15.174:0/1540119130 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89700f5410 0x7f89700f5830 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:06.650 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.647+0000 7f89757f9700 1 -- 172.21.15.174:0/1540119130 >> 172.21.15.174:0/1540119130 conn(0x7f89700f0b70 msgr2=0x7f89700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:06.650 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.647+0000 7f89757f9700 1 -- 172.21.15.174:0/1540119130 shutdown_connections 2023-12-25T21:09:06.650 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.647+0000 7f89757f9700 1 -- 172.21.15.174:0/1540119130 wait complete. 2023-12-25T21:09:06.650 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.648+0000 7f89757f9700 1 Processor -- start 2023-12-25T21:09:06.650 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.648+0000 7f89757f9700 1 -- start start 2023-12-25T21:09:06.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.648+0000 7f89757f9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89700f5410 0x7f8970109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:06.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.648+0000 7f89757f9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8970060220 con 0x7f89700f5410 2023-12-25T21:09:06.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.648+0000 7f896effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89700f5410 0x7f8970109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:06.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.648+0000 7f896effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89700f5410 0x7f8970109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:44304/0 (socket says 172.21.15.174:44304) 2023-12-25T21:09:06.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.648+0000 7f896effd700 1 -- 172.21.15.174:0/344059491 learned_addr learned my addr 172.21.15.174:0/344059491 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:06.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.649+0000 7f896effd700 1 -- 172.21.15.174:0/344059491 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8964005040 con 0x7f89700f5410 2023-12-25T21:09:06.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.649+0000 7f896effd700 1 --2- 172.21.15.174:0/344059491 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89700f5410 0x7f8970109420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f8964013510 tx=0x7f896400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:06.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.649+0000 7f89577fe700 1 -- 172.21.15.174:0/344059491 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8964009320 con 0x7f89700f5410 2023-12-25T21:09:06.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.649+0000 7f89757f9700 1 -- 172.21.15.174:0/344059491 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8970060420 con 0x7f89700f5410 2023-12-25T21:09:06.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.649+0000 7f89577fe700 1 -- 172.21.15.174:0/344059491 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8964013f00 con 0x7f89700f5410 2023-12-25T21:09:06.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.649+0000 7f89577fe700 1 -- 172.21.15.174:0/344059491 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8964014d90 con 0x7f89700f5410 2023-12-25T21:09:06.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.649+0000 7f89757f9700 1 -- 172.21.15.174:0/344059491 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8970109ad0 con 0x7f89700f5410 2023-12-25T21:09:06.652 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.650+0000 7f89577fe700 1 -- 172.21.15.174:0/344059491 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8964014420 con 0x7f89700f5410 2023-12-25T21:09:06.652 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.650+0000 7f89757f9700 1 -- 172.21.15.174:0/344059491 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8970059100 con 0x7f89700f5410 2023-12-25T21:09:06.652 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.650+0000 7f89577fe700 1 --2- 172.21.15.174:0/344059491 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f895802ff60 0x7f8958032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:06.652 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.650+0000 7f89577fe700 1 -- 172.21.15.174:0/344059491 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8964041000 con 0x7f89700f5410 2023-12-25T21:09:06.652 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.650+0000 7f896e7fc700 1 --2- 172.21.15.174:0/344059491 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f895802ff60 0x7f8958032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:06.653 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.651+0000 7f896e7fc700 1 --2- 172.21.15.174:0/344059491 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f895802ff60 0x7f8958032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f89600096a0 tx=0x7f8960006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:06.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.655+0000 7f89577fe700 1 -- 172.21.15.174:0/344059491 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f89640095f0 con 0x7f89700f5410 2023-12-25T21:09:06.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:06 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:04.587174+0000 mgr.smithi061.jxxeke (mgr.14164) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:06.861 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.860+0000 7f89757f9700 1 -- 172.21.15.174:0/344059491 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8970045ad0 con 0x7f89700f5410 2023-12-25T21:09:06.862 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.861+0000 7f89577fe700 1 -- 172.21.15.174:0/344059491 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f896401db70 con 0x7f89700f5410 2023-12-25T21:09:06.862 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:06.862 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:06.864 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.862+0000 7f89757f9700 1 -- 172.21.15.174:0/344059491 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f895802ff60 msgr2=0x7f8958032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:06.864 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.862+0000 7f89757f9700 1 --2- 172.21.15.174:0/344059491 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f895802ff60 0x7f8958032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f89600096a0 tx=0x7f8960006b40).stop 2023-12-25T21:09:06.864 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.862+0000 7f89757f9700 1 -- 172.21.15.174:0/344059491 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89700f5410 msgr2=0x7f8970109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:06.864 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.862+0000 7f89757f9700 1 --2- 172.21.15.174:0/344059491 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89700f5410 0x7f8970109420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f8964013510 tx=0x7f896400b040).stop 2023-12-25T21:09:06.865 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.862+0000 7f89757f9700 1 -- 172.21.15.174:0/344059491 shutdown_connections 2023-12-25T21:09:06.865 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.862+0000 7f89757f9700 1 --2- 172.21.15.174:0/344059491 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f895802ff60 0x7f8958032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:06.865 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.862+0000 7f89757f9700 1 --2- 172.21.15.174:0/344059491 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89700f5410 0x7f8970109420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:06.865 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.862+0000 7f89757f9700 1 -- 172.21.15.174:0/344059491 >> 172.21.15.174:0/344059491 conn(0x7f89700f0b70 msgr2=0x7f89700f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:06.865 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.862+0000 7f89757f9700 1 -- 172.21.15.174:0/344059491 shutdown_connections 2023-12-25T21:09:06.865 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:06.862+0000 7f89757f9700 1 -- 172.21.15.174:0/344059491 wait complete. 2023-12-25T21:09:06.866 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:07.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:07 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:06.862134+0000 mon.smithi061 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.174:0/344059491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:08.273 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:08.273 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:08.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:08 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:06.587555+0000 mgr.smithi061.jxxeke (mgr.14164) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:09.443 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.442+0000 7fcdf4b25700 1 -- 172.21.15.174:0/2817128638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdf00f5410 msgr2=0x7fcdf00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:09.443 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.442+0000 7fcdf4b25700 1 --2- 172.21.15.174:0/2817128638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdf00f5410 0x7fcdf00f5830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fcdd8009fb0 tx=0x7fcdd8009b30).stop 2023-12-25T21:09:09.443 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.442+0000 7fcdf4b25700 1 -- 172.21.15.174:0/2817128638 shutdown_connections 2023-12-25T21:09:09.444 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.442+0000 7fcdf4b25700 1 --2- 172.21.15.174:0/2817128638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdf00f5410 0x7fcdf00f5830 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:09.444 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.442+0000 7fcdf4b25700 1 -- 172.21.15.174:0/2817128638 >> 172.21.15.174:0/2817128638 conn(0x7fcdf00f0b70 msgr2=0x7fcdf00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:09.444 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.442+0000 7fcdf4b25700 1 -- 172.21.15.174:0/2817128638 shutdown_connections 2023-12-25T21:09:09.444 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.442+0000 7fcdf4b25700 1 -- 172.21.15.174:0/2817128638 wait complete. 2023-12-25T21:09:09.444 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.442+0000 7fcdf4b25700 1 Processor -- start 2023-12-25T21:09:09.444 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.442+0000 7fcdf4b25700 1 -- start start 2023-12-25T21:09:09.444 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.442+0000 7fcdf4b25700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdf00f5410 0x7fcdf0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:09.444 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.442+0000 7fcdf4b25700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf0060240 con 0x7fcdf00f5410 2023-12-25T21:09:09.446 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.443+0000 7fcdee59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdf00f5410 0x7fcdf0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:09.446 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.443+0000 7fcdee59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdf00f5410 0x7fcdf0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:44322/0 (socket says 172.21.15.174:44322) 2023-12-25T21:09:09.446 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.443+0000 7fcdee59c700 1 -- 172.21.15.174:0/1818414629 learned_addr learned my addr 172.21.15.174:0/1818414629 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:09.446 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.443+0000 7fcdee59c700 1 -- 172.21.15.174:0/1818414629 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd8005040 con 0x7fcdf00f5410 2023-12-25T21:09:09.446 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.443+0000 7fcdee59c700 1 --2- 172.21.15.174:0/1818414629 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdf00f5410 0x7fcdf0109430 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fcdd8013510 tx=0x7fcdd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:09.446 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.443+0000 7fcde6ffd700 1 -- 172.21.15.174:0/1818414629 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdd8009320 con 0x7fcdf00f5410 2023-12-25T21:09:09.446 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.443+0000 7fcdf4b25700 1 -- 172.21.15.174:0/1818414629 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf0060440 con 0x7fcdf00f5410 2023-12-25T21:09:09.446 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.443+0000 7fcde6ffd700 1 -- 172.21.15.174:0/1818414629 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcdd8013f00 con 0x7fcdf00f5410 2023-12-25T21:09:09.447 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.443+0000 7fcde6ffd700 1 -- 172.21.15.174:0/1818414629 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdd8014d90 con 0x7fcdf00f5410 2023-12-25T21:09:09.447 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.444+0000 7fcdf4b25700 1 -- 172.21.15.174:0/1818414629 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf0109ae0 con 0x7fcdf00f5410 2023-12-25T21:09:09.447 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.444+0000 7fcde6ffd700 1 -- 172.21.15.174:0/1818414629 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcdd8014420 con 0x7fcdf00f5410 2023-12-25T21:09:09.447 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.444+0000 7fcdf4b25700 1 -- 172.21.15.174:0/1818414629 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf0045ad0 con 0x7fcdf00f5410 2023-12-25T21:09:09.447 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.444+0000 7fcde6ffd700 1 --2- 172.21.15.174:0/1818414629 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fcddc02ff60 0x7fcddc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:09.447 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.444+0000 7fcde6ffd700 1 -- 172.21.15.174:0/1818414629 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcdd8041000 con 0x7fcdf00f5410 2023-12-25T21:09:09.447 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.445+0000 7fcdedd9b700 1 --2- 172.21.15.174:0/1818414629 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fcddc02ff60 0x7fcddc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:09.447 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.445+0000 7fcdedd9b700 1 --2- 172.21.15.174:0/1818414629 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fcddc02ff60 0x7fcddc032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fcde00096a0 tx=0x7fcde0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:09.450 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.449+0000 7fcde6ffd700 1 -- 172.21.15.174:0/1818414629 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcdd80095f0 con 0x7fcdf00f5410 2023-12-25T21:09:09.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.654+0000 7fcdf4b25700 1 -- 172.21.15.174:0/1818414629 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcdf00fae60 con 0x7fcdf00f5410 2023-12-25T21:09:09.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.655+0000 7fcde6ffd700 1 -- 172.21.15.174:0/1818414629 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcdd801db70 con 0x7fcdf00f5410 2023-12-25T21:09:09.655 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:09.656 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:09.657 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.656+0000 7fcdf4b25700 1 -- 172.21.15.174:0/1818414629 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fcddc02ff60 msgr2=0x7fcddc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:09.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.656+0000 7fcdf4b25700 1 --2- 172.21.15.174:0/1818414629 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fcddc02ff60 0x7fcddc032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fcde00096a0 tx=0x7fcde0006b40).stop 2023-12-25T21:09:09.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.656+0000 7fcdf4b25700 1 -- 172.21.15.174:0/1818414629 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdf00f5410 msgr2=0x7fcdf0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:09.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.656+0000 7fcdf4b25700 1 --2- 172.21.15.174:0/1818414629 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdf00f5410 0x7fcdf0109430 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fcdd8013510 tx=0x7fcdd800b040).stop 2023-12-25T21:09:09.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.656+0000 7fcdf4b25700 1 -- 172.21.15.174:0/1818414629 shutdown_connections 2023-12-25T21:09:09.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.656+0000 7fcdf4b25700 1 --2- 172.21.15.174:0/1818414629 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fcddc02ff60 0x7fcddc032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:09.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.656+0000 7fcdf4b25700 1 --2- 172.21.15.174:0/1818414629 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdf00f5410 0x7fcdf0109430 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:09.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.656+0000 7fcdf4b25700 1 -- 172.21.15.174:0/1818414629 >> 172.21.15.174:0/1818414629 conn(0x7fcdf00f0b70 msgr2=0x7fcdf00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:09.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.656+0000 7fcdf4b25700 1 -- 172.21.15.174:0/1818414629 shutdown_connections 2023-12-25T21:09:09.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:09.656+0000 7fcdf4b25700 1 -- 172.21.15.174:0/1818414629 wait complete. 2023-12-25T21:09:09.659 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:10.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:10 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:08.587888+0000 mgr.smithi061.jxxeke (mgr.14164) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:10.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:10 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:09.656217+0000 mon.smithi061 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.174:0/1818414629' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:11.077 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:11.078 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:12.199 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.197+0000 7f11ed4bc700 1 -- 172.21.15.174:0/2282580959 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11e80601e0 msgr2=0x7f11e80f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:12.199 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.197+0000 7f11ed4bc700 1 --2- 172.21.15.174:0/2282580959 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11e80601e0 0x7f11e80f8f20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f11d8009fb0 tx=0x7f11d8009b30).stop 2023-12-25T21:09:12.199 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.197+0000 7f11ed4bc700 1 -- 172.21.15.174:0/2282580959 shutdown_connections 2023-12-25T21:09:12.199 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.197+0000 7f11ed4bc700 1 --2- 172.21.15.174:0/2282580959 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11e80601e0 0x7f11e80f8f20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:12.199 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.197+0000 7f11ed4bc700 1 -- 172.21.15.174:0/2282580959 >> 172.21.15.174:0/2282580959 conn(0x7f11e80f09a0 msgr2=0x7f11e80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:12.199 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.197+0000 7f11ed4bc700 1 -- 172.21.15.174:0/2282580959 shutdown_connections 2023-12-25T21:09:12.199 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.197+0000 7f11ed4bc700 1 -- 172.21.15.174:0/2282580959 wait complete. 2023-12-25T21:09:12.199 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.198+0000 7f11ed4bc700 1 Processor -- start 2023-12-25T21:09:12.199 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.198+0000 7f11ed4bc700 1 -- start start 2023-12-25T21:09:12.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.199+0000 7f11ed4bc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11e80601e0 0x7f11e8106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:12.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.199+0000 7f11ed4bc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e81069f0 con 0x7f11e80601e0 2023-12-25T21:09:12.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.199+0000 7f11e7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11e80601e0 0x7f11e8106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:12.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.199+0000 7f11e7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11e80601e0 0x7f11e8106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:44346/0 (socket says 172.21.15.174:44346) 2023-12-25T21:09:12.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.199+0000 7f11e7fff700 1 -- 172.21.15.174:0/1347251994 learned_addr learned my addr 172.21.15.174:0/1347251994 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:12.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.199+0000 7f11e7fff700 1 -- 172.21.15.174:0/1347251994 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d8005040 con 0x7f11e80601e0 2023-12-25T21:09:12.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.199+0000 7f11e7fff700 1 --2- 172.21.15.174:0/1347251994 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11e80601e0 0x7f11e8106500 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f11d8013510 tx=0x7f11d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:12.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.199+0000 7f11e4ff9700 1 -- 172.21.15.174:0/1347251994 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11d8009320 con 0x7f11e80601e0 2023-12-25T21:09:12.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.199+0000 7f11ed4bc700 1 -- 172.21.15.174:0/1347251994 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e8106bf0 con 0x7f11e80601e0 2023-12-25T21:09:12.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.199+0000 7f11e4ff9700 1 -- 172.21.15.174:0/1347251994 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11d8013f00 con 0x7f11e80601e0 2023-12-25T21:09:12.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.199+0000 7f11e4ff9700 1 -- 172.21.15.174:0/1347251994 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11d8014d90 con 0x7f11e80601e0 2023-12-25T21:09:12.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.199+0000 7f11ed4bc700 1 -- 172.21.15.174:0/1347251994 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e8107080 con 0x7f11e80601e0 2023-12-25T21:09:12.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.200+0000 7f11e4ff9700 1 -- 172.21.15.174:0/1347251994 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f11d80095f0 con 0x7f11e80601e0 2023-12-25T21:09:12.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.200+0000 7f11ed4bc700 1 -- 172.21.15.174:0/1347251994 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e8045ad0 con 0x7f11e80601e0 2023-12-25T21:09:12.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.200+0000 7f11e4ff9700 1 --2- 172.21.15.174:0/1347251994 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f11d002ff60 0x7f11d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:12.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.200+0000 7f11e4ff9700 1 -- 172.21.15.174:0/1347251994 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f11d8041000 con 0x7f11e80601e0 2023-12-25T21:09:12.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.201+0000 7f11e77fe700 1 --2- 172.21.15.174:0/1347251994 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f11d002ff60 0x7f11d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:12.203 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.201+0000 7f11e77fe700 1 --2- 172.21.15.174:0/1347251994 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f11d002ff60 0x7f11d0032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f11dc0096a0 tx=0x7f11dc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:12.205 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.205+0000 7f11e4ff9700 1 -- 172.21.15.174:0/1347251994 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f11d801d7f0 con 0x7f11e80601e0 2023-12-25T21:09:12.415 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.414+0000 7f11ed4bc700 1 -- 172.21.15.174:0/1347251994 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f11e8107f30 con 0x7f11e80601e0 2023-12-25T21:09:12.416 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.415+0000 7f11e4ff9700 1 -- 172.21.15.174:0/1347251994 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f11d8010230 con 0x7f11e80601e0 2023-12-25T21:09:12.416 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:12.416 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:12.418 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.417+0000 7f11ed4bc700 1 -- 172.21.15.174:0/1347251994 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f11d002ff60 msgr2=0x7f11d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:12.418 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.417+0000 7f11ed4bc700 1 --2- 172.21.15.174:0/1347251994 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f11d002ff60 0x7f11d0032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f11dc0096a0 tx=0x7f11dc006b40).stop 2023-12-25T21:09:12.418 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.417+0000 7f11ed4bc700 1 -- 172.21.15.174:0/1347251994 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11e80601e0 msgr2=0x7f11e8106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:12.418 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.417+0000 7f11ed4bc700 1 --2- 172.21.15.174:0/1347251994 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11e80601e0 0x7f11e8106500 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f11d8013510 tx=0x7f11d800b040).stop 2023-12-25T21:09:12.418 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.417+0000 7f11ed4bc700 1 -- 172.21.15.174:0/1347251994 shutdown_connections 2023-12-25T21:09:12.418 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.417+0000 7f11ed4bc700 1 --2- 172.21.15.174:0/1347251994 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f11d002ff60 0x7f11d0032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:12.418 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.417+0000 7f11ed4bc700 1 --2- 172.21.15.174:0/1347251994 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11e80601e0 0x7f11e8106500 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:12.418 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.417+0000 7f11ed4bc700 1 -- 172.21.15.174:0/1347251994 >> 172.21.15.174:0/1347251994 conn(0x7f11e80f09a0 msgr2=0x7f11e80f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:12.418 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.417+0000 7f11ed4bc700 1 -- 172.21.15.174:0/1347251994 shutdown_connections 2023-12-25T21:09:12.418 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:12.417+0000 7f11ed4bc700 1 -- 172.21.15.174:0/1347251994 wait complete. 2023-12-25T21:09:12.419 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:12.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:12 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:10.588202+0000 mgr.smithi061.jxxeke (mgr.14164) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:13.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:13 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:12.416434+0000 mon.smithi061 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.174:0/1347251994' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:13.801 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:13.801 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:14.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:14 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:12.588516+0000 mgr.smithi061.jxxeke (mgr.14164) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:14.934 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.933+0000 7f73fa552700 1 -- 172.21.15.174:0/830766510 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73f40f5410 msgr2=0x7f73f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:14.935 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.933+0000 7f73fa552700 1 --2- 172.21.15.174:0/830766510 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73f40f5410 0x7f73f40f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f73dc009fb0 tx=0x7f73dc009b30).stop 2023-12-25T21:09:14.935 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.933+0000 7f73fa552700 1 -- 172.21.15.174:0/830766510 shutdown_connections 2023-12-25T21:09:14.935 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.933+0000 7f73fa552700 1 --2- 172.21.15.174:0/830766510 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73f40f5410 0x7f73f40f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:14.935 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.933+0000 7f73fa552700 1 -- 172.21.15.174:0/830766510 >> 172.21.15.174:0/830766510 conn(0x7f73f40f0b70 msgr2=0x7f73f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:14.935 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.933+0000 7f73fa552700 1 -- 172.21.15.174:0/830766510 shutdown_connections 2023-12-25T21:09:14.935 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.933+0000 7f73fa552700 1 -- 172.21.15.174:0/830766510 wait complete. 2023-12-25T21:09:14.935 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.934+0000 7f73fa552700 1 Processor -- start 2023-12-25T21:09:14.935 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.934+0000 7f73fa552700 1 -- start start 2023-12-25T21:09:14.937 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.934+0000 7f73fa552700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73f40f5410 0x7f73f4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:14.937 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.934+0000 7f73fa552700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f4105b20 con 0x7f73f40f5410 2023-12-25T21:09:14.937 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.934+0000 7f73f3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73f40f5410 0x7f73f4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:14.937 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.935+0000 7f73f3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73f40f5410 0x7f73f4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:44370/0 (socket says 172.21.15.174:44370) 2023-12-25T21:09:14.937 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.935+0000 7f73f3fff700 1 -- 172.21.15.174:0/2556745343 learned_addr learned my addr 172.21.15.174:0/2556745343 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:14.937 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.935+0000 7f73f3fff700 1 -- 172.21.15.174:0/2556745343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73dc005040 con 0x7f73f40f5410 2023-12-25T21:09:14.937 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.935+0000 7f73f3fff700 1 --2- 172.21.15.174:0/2556745343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73f40f5410 0x7f73f4105630 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f73dc013510 tx=0x7f73dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:14.937 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.935+0000 7f73f0ff9700 1 -- 172.21.15.174:0/2556745343 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73dc009320 con 0x7f73f40f5410 2023-12-25T21:09:14.937 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.935+0000 7f73fa552700 1 -- 172.21.15.174:0/2556745343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f4105d20 con 0x7f73f40f5410 2023-12-25T21:09:14.937 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.935+0000 7f73f0ff9700 1 -- 172.21.15.174:0/2556745343 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73dc013f00 con 0x7f73f40f5410 2023-12-25T21:09:14.938 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.935+0000 7f73f0ff9700 1 -- 172.21.15.174:0/2556745343 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73dc014d90 con 0x7f73f40f5410 2023-12-25T21:09:14.938 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.935+0000 7f73fa552700 1 -- 172.21.15.174:0/2556745343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f41061b0 con 0x7f73f40f5410 2023-12-25T21:09:14.938 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.936+0000 7f73f0ff9700 1 -- 172.21.15.174:0/2556745343 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f73dc0095f0 con 0x7f73f40f5410 2023-12-25T21:09:14.938 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.936+0000 7f73fa552700 1 -- 172.21.15.174:0/2556745343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f4045ad0 con 0x7f73f40f5410 2023-12-25T21:09:14.938 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.936+0000 7f73f0ff9700 1 --2- 172.21.15.174:0/2556745343 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f73e002ff60 0x7f73e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:14.938 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.936+0000 7f73f0ff9700 1 -- 172.21.15.174:0/2556745343 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f73dc0410c0 con 0x7f73f40f5410 2023-12-25T21:09:14.938 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.936+0000 7f73f37fe700 1 --2- 172.21.15.174:0/2556745343 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f73e002ff60 0x7f73e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:14.938 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.937+0000 7f73f37fe700 1 --2- 172.21.15.174:0/2556745343 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f73e002ff60 0x7f73e0032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f73e40096a0 tx=0x7f73e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:14.942 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:14.941+0000 7f73f0ff9700 1 -- 172.21.15.174:0/2556745343 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f73dc014420 con 0x7f73f40f5410 2023-12-25T21:09:15.148 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:15.148+0000 7f73fa552700 1 -- 172.21.15.174:0/2556745343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f73f40fae60 con 0x7f73f40f5410 2023-12-25T21:09:15.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:15.148+0000 7f73f0ff9700 1 -- 172.21.15.174:0/2556745343 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f73dc010230 con 0x7f73f40f5410 2023-12-25T21:09:15.150 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:15.150 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:15.151 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:15.150+0000 7f73fa552700 1 -- 172.21.15.174:0/2556745343 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f73e002ff60 msgr2=0x7f73e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:15.151 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:15.150+0000 7f73fa552700 1 --2- 172.21.15.174:0/2556745343 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f73e002ff60 0x7f73e0032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f73e40096a0 tx=0x7f73e4006b40).stop 2023-12-25T21:09:15.151 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:15.150+0000 7f73fa552700 1 -- 172.21.15.174:0/2556745343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73f40f5410 msgr2=0x7f73f4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:15.151 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:15.150+0000 7f73fa552700 1 --2- 172.21.15.174:0/2556745343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73f40f5410 0x7f73f4105630 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f73dc013510 tx=0x7f73dc00b040).stop 2023-12-25T21:09:15.151 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:15.150+0000 7f73fa552700 1 -- 172.21.15.174:0/2556745343 shutdown_connections 2023-12-25T21:09:15.151 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:15.150+0000 7f73fa552700 1 --2- 172.21.15.174:0/2556745343 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f73e002ff60 0x7f73e0032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:15.152 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:15.150+0000 7f73fa552700 1 --2- 172.21.15.174:0/2556745343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73f40f5410 0x7f73f4105630 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:15.152 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:15.150+0000 7f73fa552700 1 -- 172.21.15.174:0/2556745343 >> 172.21.15.174:0/2556745343 conn(0x7f73f40f0b70 msgr2=0x7f73f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:15.152 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:15.150+0000 7f73fa552700 1 -- 172.21.15.174:0/2556745343 shutdown_connections 2023-12-25T21:09:15.152 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:15.150+0000 7f73fa552700 1 -- 172.21.15.174:0/2556745343 wait complete. 2023-12-25T21:09:15.152 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:15.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:15 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:15.149803+0000 mon.smithi061 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.174:0/2556745343' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:16.532 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:16.533 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:16.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:16 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:14.588797+0000 mgr.smithi061.jxxeke (mgr.14164) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:17.597 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.596+0000 7fd4a971b700 1 -- 172.21.15.174:0/635418470 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4a40f62a0 msgr2=0x7fd4a40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:17.597 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.596+0000 7fd4a971b700 1 --2- 172.21.15.174:0/635418470 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4a40f62a0 0x7fd4a40f66c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fd494009fb0 tx=0x7fd494009b30).stop 2023-12-25T21:09:17.597 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.596+0000 7fd4a971b700 1 -- 172.21.15.174:0/635418470 shutdown_connections 2023-12-25T21:09:17.597 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.596+0000 7fd4a971b700 1 --2- 172.21.15.174:0/635418470 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4a40f62a0 0x7fd4a40f66c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:17.597 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.596+0000 7fd4a971b700 1 -- 172.21.15.174:0/635418470 >> 172.21.15.174:0/635418470 conn(0x7fd4a40f1aa0 msgr2=0x7fd4a40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:17.597 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.596+0000 7fd4a971b700 1 -- 172.21.15.174:0/635418470 shutdown_connections 2023-12-25T21:09:17.598 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.596+0000 7fd4a971b700 1 -- 172.21.15.174:0/635418470 wait complete. 2023-12-25T21:09:17.598 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.596+0000 7fd4a971b700 1 Processor -- start 2023-12-25T21:09:17.598 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.596+0000 7fd4a971b700 1 -- start start 2023-12-25T21:09:17.598 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.596+0000 7fd4a971b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4a40f62a0 0x7fd4a41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:17.598 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.596+0000 7fd4a971b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a41069a0 con 0x7fd4a40f62a0 2023-12-25T21:09:17.599 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.597+0000 7fd4a3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4a40f62a0 0x7fd4a41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:17.600 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.597+0000 7fd4a3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4a40f62a0 0x7fd4a41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:54422/0 (socket says 172.21.15.174:54422) 2023-12-25T21:09:17.600 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.597+0000 7fd4a3fff700 1 -- 172.21.15.174:0/2618626798 learned_addr learned my addr 172.21.15.174:0/2618626798 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:17.600 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.597+0000 7fd4a3fff700 1 -- 172.21.15.174:0/2618626798 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd494005040 con 0x7fd4a40f62a0 2023-12-25T21:09:17.600 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.597+0000 7fd4a3fff700 1 --2- 172.21.15.174:0/2618626798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4a40f62a0 0x7fd4a41064b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fd494013510 tx=0x7fd49400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:17.600 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.597+0000 7fd4a0ff9700 1 -- 172.21.15.174:0/2618626798 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd494009320 con 0x7fd4a40f62a0 2023-12-25T21:09:17.600 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.597+0000 7fd4a971b700 1 -- 172.21.15.174:0/2618626798 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a4106ba0 con 0x7fd4a40f62a0 2023-12-25T21:09:17.600 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.597+0000 7fd4a0ff9700 1 -- 172.21.15.174:0/2618626798 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd494013f00 con 0x7fd4a40f62a0 2023-12-25T21:09:17.600 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.597+0000 7fd4a0ff9700 1 -- 172.21.15.174:0/2618626798 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd494014dc0 con 0x7fd4a40f62a0 2023-12-25T21:09:17.600 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.597+0000 7fd4a971b700 1 -- 172.21.15.174:0/2618626798 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a4107030 con 0x7fd4a40f62a0 2023-12-25T21:09:17.600 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.598+0000 7fd4a0ff9700 1 -- 172.21.15.174:0/2618626798 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd49401d3d0 con 0x7fd4a40f62a0 2023-12-25T21:09:17.600 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.598+0000 7fd4a0ff9700 1 --2- 172.21.15.174:0/2618626798 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd48c02ff60 0x7fd48c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:17.600 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.598+0000 7fd4a0ff9700 1 -- 172.21.15.174:0/2618626798 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd4940410d0 con 0x7fd4a40f62a0 2023-12-25T21:09:17.601 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.598+0000 7fd4a971b700 1 -- 172.21.15.174:0/2618626798 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a4045ad0 con 0x7fd4a40f62a0 2023-12-25T21:09:17.601 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.599+0000 7fd4a37fe700 1 --2- 172.21.15.174:0/2618626798 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd48c02ff60 0x7fd48c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:17.601 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.599+0000 7fd4a37fe700 1 --2- 172.21.15.174:0/2618626798 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd48c02ff60 0x7fd48c032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fd4980096a0 tx=0x7fd498006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:17.604 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.603+0000 7fd4a0ff9700 1 -- 172.21.15.174:0/2618626798 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd494012900 con 0x7fd4a40f62a0 2023-12-25T21:09:17.809 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.808+0000 7fd4a971b700 1 -- 172.21.15.174:0/2618626798 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4a4107d50 con 0x7fd4a40f62a0 2023-12-25T21:09:17.810 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.809+0000 7fd4a0ff9700 1 -- 172.21.15.174:0/2618626798 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd49402b020 con 0x7fd4a40f62a0 2023-12-25T21:09:17.810 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:17.810 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:17.812 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.811+0000 7fd4a971b700 1 -- 172.21.15.174:0/2618626798 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd48c02ff60 msgr2=0x7fd48c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:17.812 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.811+0000 7fd4a971b700 1 --2- 172.21.15.174:0/2618626798 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd48c02ff60 0x7fd48c032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fd4980096a0 tx=0x7fd498006b40).stop 2023-12-25T21:09:17.812 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.811+0000 7fd4a971b700 1 -- 172.21.15.174:0/2618626798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4a40f62a0 msgr2=0x7fd4a41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:17.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.811+0000 7fd4a971b700 1 --2- 172.21.15.174:0/2618626798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4a40f62a0 0x7fd4a41064b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fd494013510 tx=0x7fd49400b040).stop 2023-12-25T21:09:17.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.811+0000 7fd4a971b700 1 -- 172.21.15.174:0/2618626798 shutdown_connections 2023-12-25T21:09:17.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.811+0000 7fd4a971b700 1 --2- 172.21.15.174:0/2618626798 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd48c02ff60 0x7fd48c032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:17.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.811+0000 7fd4a971b700 1 --2- 172.21.15.174:0/2618626798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4a40f62a0 0x7fd4a41064b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:17.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.811+0000 7fd4a971b700 1 -- 172.21.15.174:0/2618626798 >> 172.21.15.174:0/2618626798 conn(0x7fd4a40f1aa0 msgr2=0x7fd4a40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:17.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.811+0000 7fd4a971b700 1 -- 172.21.15.174:0/2618626798 shutdown_connections 2023-12-25T21:09:17.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:17.811+0000 7fd4a971b700 1 -- 172.21.15.174:0/2618626798 wait complete. 2023-12-25T21:09:17.814 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:18.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:18 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:16.589117+0000 mgr.smithi061.jxxeke (mgr.14164) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:18.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:18 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:17.810301+0000 mon.smithi061 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.174:0/2618626798' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:19.314 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:19.315 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:20.445 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.444+0000 7f715523a700 1 -- 172.21.15.174:0/3609536707 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500f6510 msgr2=0x7f71500f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:20.445 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.444+0000 7f715523a700 1 --2- 172.21.15.174:0/3609536707 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500f6510 0x7f71500f6930 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f7138009fb0 tx=0x7f7138009b30).stop 2023-12-25T21:09:20.445 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.444+0000 7f715523a700 1 -- 172.21.15.174:0/3609536707 shutdown_connections 2023-12-25T21:09:20.445 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.444+0000 7f715523a700 1 --2- 172.21.15.174:0/3609536707 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500f6510 0x7f71500f6930 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:20.445 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.444+0000 7f715523a700 1 -- 172.21.15.174:0/3609536707 >> 172.21.15.174:0/3609536707 conn(0x7f71500f1c70 msgr2=0x7f71500f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:20.446 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.444+0000 7f715523a700 1 -- 172.21.15.174:0/3609536707 shutdown_connections 2023-12-25T21:09:20.446 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.444+0000 7f715523a700 1 -- 172.21.15.174:0/3609536707 wait complete. 2023-12-25T21:09:20.447 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.445+0000 7f715523a700 1 Processor -- start 2023-12-25T21:09:20.447 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.445+0000 7f715523a700 1 -- start start 2023-12-25T21:09:20.448 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.445+0000 7f715523a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500f6510 0x7f715006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:20.448 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.445+0000 7f715523a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7150069e20 con 0x7f71500f6510 2023-12-25T21:09:20.448 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.445+0000 7f714ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500f6510 0x7f715006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:20.448 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.445+0000 7f714ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500f6510 0x7f715006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:54442/0 (socket says 172.21.15.174:54442) 2023-12-25T21:09:20.448 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.445+0000 7f714ed9d700 1 -- 172.21.15.174:0/431039230 learned_addr learned my addr 172.21.15.174:0/431039230 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:20.448 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.446+0000 7f714ed9d700 1 -- 172.21.15.174:0/431039230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7138005040 con 0x7f71500f6510 2023-12-25T21:09:20.448 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.446+0000 7f714ed9d700 1 --2- 172.21.15.174:0/431039230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500f6510 0x7f715006b9a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f7138013510 tx=0x7f713800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:20.448 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.446+0000 7f71477fe700 1 -- 172.21.15.174:0/431039230 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7138009320 con 0x7f71500f6510 2023-12-25T21:09:20.448 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.446+0000 7f715523a700 1 -- 172.21.15.174:0/431039230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7150069fc0 con 0x7f71500f6510 2023-12-25T21:09:20.448 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.446+0000 7f71477fe700 1 -- 172.21.15.174:0/431039230 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7138013f00 con 0x7f71500f6510 2023-12-25T21:09:20.448 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.446+0000 7f71477fe700 1 -- 172.21.15.174:0/431039230 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7138014d90 con 0x7f71500f6510 2023-12-25T21:09:20.448 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.446+0000 7f715523a700 1 -- 172.21.15.174:0/431039230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f715006a450 con 0x7f71500f6510 2023-12-25T21:09:20.448 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.447+0000 7f71477fe700 1 -- 172.21.15.174:0/431039230 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71380095f0 con 0x7f71500f6510 2023-12-25T21:09:20.449 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.447+0000 7f715523a700 1 -- 172.21.15.174:0/431039230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7150059100 con 0x7f71500f6510 2023-12-25T21:09:20.449 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.447+0000 7f71477fe700 1 --2- 172.21.15.174:0/431039230 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f713c02ff60 0x7f713c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:20.449 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.447+0000 7f71477fe700 1 -- 172.21.15.174:0/431039230 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7138041080 con 0x7f71500f6510 2023-12-25T21:09:20.449 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.447+0000 7f714e59c700 1 --2- 172.21.15.174:0/431039230 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f713c02ff60 0x7f713c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:20.449 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.447+0000 7f714e59c700 1 --2- 172.21.15.174:0/431039230 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f713c02ff60 0x7f713c032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f71400096a0 tx=0x7f7140006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:20.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.451+0000 7f71477fe700 1 -- 172.21.15.174:0/431039230 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7138014420 con 0x7f71500f6510 2023-12-25T21:09:20.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.657+0000 7f715523a700 1 -- 172.21.15.174:0/431039230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7150045ad0 con 0x7f71500f6510 2023-12-25T21:09:20.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.658+0000 7f71477fe700 1 -- 172.21.15.174:0/431039230 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7138010230 con 0x7f71500f6510 2023-12-25T21:09:20.659 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:20.659 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:20.660 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.660+0000 7f715523a700 1 -- 172.21.15.174:0/431039230 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f713c02ff60 msgr2=0x7f713c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:20.660 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.660+0000 7f715523a700 1 --2- 172.21.15.174:0/431039230 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f713c02ff60 0x7f713c032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f71400096a0 tx=0x7f7140006b40).stop 2023-12-25T21:09:20.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.660+0000 7f715523a700 1 -- 172.21.15.174:0/431039230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500f6510 msgr2=0x7f715006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:20.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.660+0000 7f715523a700 1 --2- 172.21.15.174:0/431039230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500f6510 0x7f715006b9a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f7138013510 tx=0x7f713800b040).stop 2023-12-25T21:09:20.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.660+0000 7f715523a700 1 -- 172.21.15.174:0/431039230 shutdown_connections 2023-12-25T21:09:20.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.660+0000 7f715523a700 1 --2- 172.21.15.174:0/431039230 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f713c02ff60 0x7f713c032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:20.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.660+0000 7f715523a700 1 --2- 172.21.15.174:0/431039230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500f6510 0x7f715006b9a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:20.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.660+0000 7f715523a700 1 -- 172.21.15.174:0/431039230 >> 172.21.15.174:0/431039230 conn(0x7f71500f1c70 msgr2=0x7f71500f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:20.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.660+0000 7f715523a700 1 -- 172.21.15.174:0/431039230 shutdown_connections 2023-12-25T21:09:20.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:20.660+0000 7f715523a700 1 -- 172.21.15.174:0/431039230 wait complete. 2023-12-25T21:09:20.661 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:20.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:20 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:18.589426+0000 mgr.smithi061.jxxeke (mgr.14164) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:21.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:21 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:20.659495+0000 mon.smithi061 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.174:0/431039230' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:22.032 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:22.033 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:22.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:22 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:20.589734+0000 mgr.smithi061.jxxeke (mgr.14164) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:23.252 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.250+0000 7fecfffff700 1 -- 172.21.15.174:0/300501501 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed00060140 msgr2=0x7fed000f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:23.252 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.250+0000 7fecfffff700 1 --2- 172.21.15.174:0/300501501 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed00060140 0x7fed000f8fc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fecf0009fb0 tx=0x7fecf0009b30).stop 2023-12-25T21:09:23.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.250+0000 7fecfffff700 1 -- 172.21.15.174:0/300501501 shutdown_connections 2023-12-25T21:09:23.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.250+0000 7fecfffff700 1 --2- 172.21.15.174:0/300501501 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed00060140 0x7fed000f8fc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:23.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.250+0000 7fecfffff700 1 -- 172.21.15.174:0/300501501 >> 172.21.15.174:0/300501501 conn(0x7fed000f09a0 msgr2=0x7fed000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:23.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.250+0000 7fecfffff700 1 -- 172.21.15.174:0/300501501 shutdown_connections 2023-12-25T21:09:23.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.250+0000 7fecfffff700 1 -- 172.21.15.174:0/300501501 wait complete. 2023-12-25T21:09:23.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.250+0000 7fecfffff700 1 Processor -- start 2023-12-25T21:09:23.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.250+0000 7fecfffff700 1 -- start start 2023-12-25T21:09:23.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.251+0000 7fecfffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed00060140 0x7fed00106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:23.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.251+0000 7fecfffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed00106a10 con 0x7fed00060140 2023-12-25T21:09:23.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.251+0000 7fecfeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed00060140 0x7fed00106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:23.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.251+0000 7fecfeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed00060140 0x7fed00106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:54454/0 (socket says 172.21.15.174:54454) 2023-12-25T21:09:23.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.251+0000 7fecfeffd700 1 -- 172.21.15.174:0/2087322877 learned_addr learned my addr 172.21.15.174:0/2087322877 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:23.254 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.251+0000 7fecfeffd700 1 -- 172.21.15.174:0/2087322877 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf0005040 con 0x7fed00060140 2023-12-25T21:09:23.254 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.251+0000 7fecfeffd700 1 --2- 172.21.15.174:0/2087322877 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed00060140 0x7fed00106520 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fecf0013510 tx=0x7fecf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:23.254 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.251+0000 7fece77fe700 1 -- 172.21.15.174:0/2087322877 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecf0009320 con 0x7fed00060140 2023-12-25T21:09:23.254 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.252+0000 7fecfffff700 1 -- 172.21.15.174:0/2087322877 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed00106c10 con 0x7fed00060140 2023-12-25T21:09:23.254 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.252+0000 7fece77fe700 1 -- 172.21.15.174:0/2087322877 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fecf0013f00 con 0x7fed00060140 2023-12-25T21:09:23.254 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.252+0000 7fece77fe700 1 -- 172.21.15.174:0/2087322877 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecf0014d90 con 0x7fed00060140 2023-12-25T21:09:23.254 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.252+0000 7fecfffff700 1 -- 172.21.15.174:0/2087322877 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed001070a0 con 0x7fed00060140 2023-12-25T21:09:23.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.252+0000 7fece77fe700 1 -- 172.21.15.174:0/2087322877 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fecf0014420 con 0x7fed00060140 2023-12-25T21:09:23.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.252+0000 7fecfffff700 1 -- 172.21.15.174:0/2087322877 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed00045ad0 con 0x7fed00060140 2023-12-25T21:09:23.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.252+0000 7fece77fe700 1 --2- 172.21.15.174:0/2087322877 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fece802ff60 0x7fece8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:23.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.253+0000 7fece77fe700 1 -- 172.21.15.174:0/2087322877 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fecf0040ff0 con 0x7fed00060140 2023-12-25T21:09:23.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.253+0000 7fecfe7fc700 1 --2- 172.21.15.174:0/2087322877 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fece802ff60 0x7fece8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:23.256 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.253+0000 7fecfe7fc700 1 --2- 172.21.15.174:0/2087322877 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fece802ff60 0x7fece8032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fecf40096a0 tx=0x7fecf4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:23.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.257+0000 7fece77fe700 1 -- 172.21.15.174:0/2087322877 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fecf00095f0 con 0x7fed00060140 2023-12-25T21:09:23.462 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.460+0000 7fecfffff700 1 -- 172.21.15.174:0/2087322877 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed00107f50 con 0x7fed00060140 2023-12-25T21:09:23.465 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.461+0000 7fece77fe700 1 -- 172.21.15.174:0/2087322877 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fecf001db70 con 0x7fed00060140 2023-12-25T21:09:23.466 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:23.466 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:23.467 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.462+0000 7fecfffff700 1 -- 172.21.15.174:0/2087322877 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fece802ff60 msgr2=0x7fece8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:23.467 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.462+0000 7fecfffff700 1 --2- 172.21.15.174:0/2087322877 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fece802ff60 0x7fece8032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fecf40096a0 tx=0x7fecf4006b40).stop 2023-12-25T21:09:23.467 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.462+0000 7fecfffff700 1 -- 172.21.15.174:0/2087322877 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed00060140 msgr2=0x7fed00106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:23.467 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.462+0000 7fecfffff700 1 --2- 172.21.15.174:0/2087322877 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed00060140 0x7fed00106520 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fecf0013510 tx=0x7fecf000b040).stop 2023-12-25T21:09:23.467 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.462+0000 7fecfffff700 1 -- 172.21.15.174:0/2087322877 shutdown_connections 2023-12-25T21:09:23.467 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.462+0000 7fecfffff700 1 --2- 172.21.15.174:0/2087322877 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fece802ff60 0x7fece8032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:23.467 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.463+0000 7fecfffff700 1 --2- 172.21.15.174:0/2087322877 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed00060140 0x7fed00106520 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:23.467 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.463+0000 7fecfffff700 1 -- 172.21.15.174:0/2087322877 >> 172.21.15.174:0/2087322877 conn(0x7fed000f09a0 msgr2=0x7fed000f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:23.467 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.463+0000 7fecfffff700 1 -- 172.21.15.174:0/2087322877 shutdown_connections 2023-12-25T21:09:23.468 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:23.463+0000 7fecfffff700 1 -- 172.21.15.174:0/2087322877 wait complete. 2023-12-25T21:09:23.468 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:24.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:24 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:22.590121+0000 mgr.smithi061.jxxeke (mgr.14164) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:24.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:24 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:23.462282+0000 mon.smithi061 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.174:0/2087322877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:24.870 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:24.870 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:26.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.040+0000 7f77c7e2a700 1 -- 172.21.15.174:0/2528206368 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77c00f5410 msgr2=0x7f77c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:26.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.040+0000 7f77c7e2a700 1 --2- 172.21.15.174:0/2528206368 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77c00f5410 0x7f77c00f5830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f77b0009fb0 tx=0x7f77b0009b30).stop 2023-12-25T21:09:26.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.040+0000 7f77c7e2a700 1 -- 172.21.15.174:0/2528206368 shutdown_connections 2023-12-25T21:09:26.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.040+0000 7f77c7e2a700 1 --2- 172.21.15.174:0/2528206368 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77c00f5410 0x7f77c00f5830 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:26.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.040+0000 7f77c7e2a700 1 -- 172.21.15.174:0/2528206368 >> 172.21.15.174:0/2528206368 conn(0x7f77c00f0b70 msgr2=0x7f77c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:26.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.040+0000 7f77c7e2a700 1 -- 172.21.15.174:0/2528206368 shutdown_connections 2023-12-25T21:09:26.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.040+0000 7f77c7e2a700 1 -- 172.21.15.174:0/2528206368 wait complete. 2023-12-25T21:09:26.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.041+0000 7f77c7e2a700 1 Processor -- start 2023-12-25T21:09:26.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.041+0000 7f77c7e2a700 1 -- start start 2023-12-25T21:09:26.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.041+0000 7f77c7e2a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77c00f5410 0x7f77c0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:26.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.041+0000 7f77c7e2a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c0105b20 con 0x7f77c00f5410 2023-12-25T21:09:26.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.041+0000 7f77c5bc6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77c00f5410 0x7f77c0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:26.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.041+0000 7f77c5bc6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77c00f5410 0x7f77c0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:54476/0 (socket says 172.21.15.174:54476) 2023-12-25T21:09:26.044 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.041+0000 7f77c5bc6700 1 -- 172.21.15.174:0/4153437202 learned_addr learned my addr 172.21.15.174:0/4153437202 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:26.044 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.041+0000 7f77c5bc6700 1 -- 172.21.15.174:0/4153437202 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b0005040 con 0x7f77c00f5410 2023-12-25T21:09:26.044 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.041+0000 7f77c5bc6700 1 --2- 172.21.15.174:0/4153437202 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77c00f5410 0x7f77c0105630 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f77b0014000 tx=0x7f77b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:26.044 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.042+0000 7f77b67fc700 1 -- 172.21.15.174:0/4153437202 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77b00095f0 con 0x7f77c00f5410 2023-12-25T21:09:26.044 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.042+0000 7f77c7e2a700 1 -- 172.21.15.174:0/4153437202 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c0105d20 con 0x7f77c00f5410 2023-12-25T21:09:26.044 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.042+0000 7f77b67fc700 1 -- 172.21.15.174:0/4153437202 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f77b0007e70 con 0x7f77c00f5410 2023-12-25T21:09:26.044 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.042+0000 7f77b67fc700 1 -- 172.21.15.174:0/4153437202 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77b0014ea0 con 0x7f77c00f5410 2023-12-25T21:09:26.044 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.042+0000 7f77c7e2a700 1 -- 172.21.15.174:0/4153437202 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77c01061b0 con 0x7f77c00f5410 2023-12-25T21:09:26.045 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.042+0000 7f77b67fc700 1 -- 172.21.15.174:0/4153437202 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f77b001d4e0 con 0x7f77c00f5410 2023-12-25T21:09:26.045 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.043+0000 7f77b67fc700 1 --2- 172.21.15.174:0/4153437202 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f77ac02ff60 0x7f77ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:26.046 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.043+0000 7f77b67fc700 1 -- 172.21.15.174:0/4153437202 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f77b0040ba0 con 0x7f77c00f5410 2023-12-25T21:09:26.046 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.043+0000 7f77c7e2a700 1 -- 172.21.15.174:0/4153437202 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77c0059100 con 0x7f77c00f5410 2023-12-25T21:09:26.046 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.043+0000 7f77c53c5700 1 --2- 172.21.15.174:0/4153437202 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f77ac02ff60 0x7f77ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:26.046 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.043+0000 7f77c53c5700 1 --2- 172.21.15.174:0/4153437202 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f77ac02ff60 0x7f77ac032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f77bc0096a0 tx=0x7f77bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:26.048 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.047+0000 7f77b67fc700 1 -- 172.21.15.174:0/4153437202 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f77b0025d50 con 0x7f77c00f5410 2023-12-25T21:09:26.252 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.252+0000 7f77c7e2a700 1 -- 172.21.15.174:0/4153437202 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f77c00fae60 con 0x7f77c00f5410 2023-12-25T21:09:26.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.252+0000 7f77b67fc700 1 -- 172.21.15.174:0/4153437202 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f77b0012330 con 0x7f77c00f5410 2023-12-25T21:09:26.253 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:26.253 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:26.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.254+0000 7f77c7e2a700 1 -- 172.21.15.174:0/4153437202 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f77ac02ff60 msgr2=0x7f77ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:26.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.254+0000 7f77c7e2a700 1 --2- 172.21.15.174:0/4153437202 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f77ac02ff60 0x7f77ac032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f77bc0096a0 tx=0x7f77bc006b40).stop 2023-12-25T21:09:26.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.254+0000 7f77c7e2a700 1 -- 172.21.15.174:0/4153437202 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77c00f5410 msgr2=0x7f77c0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:26.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.254+0000 7f77c7e2a700 1 --2- 172.21.15.174:0/4153437202 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77c00f5410 0x7f77c0105630 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f77b0014000 tx=0x7f77b000b040).stop 2023-12-25T21:09:26.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.254+0000 7f77c7e2a700 1 -- 172.21.15.174:0/4153437202 shutdown_connections 2023-12-25T21:09:26.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.254+0000 7f77c7e2a700 1 --2- 172.21.15.174:0/4153437202 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f77ac02ff60 0x7f77ac032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:26.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.254+0000 7f77c7e2a700 1 --2- 172.21.15.174:0/4153437202 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77c00f5410 0x7f77c0105630 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:26.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.254+0000 7f77c7e2a700 1 -- 172.21.15.174:0/4153437202 >> 172.21.15.174:0/4153437202 conn(0x7f77c00f0b70 msgr2=0x7f77c00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:26.256 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.254+0000 7f77c7e2a700 1 -- 172.21.15.174:0/4153437202 shutdown_connections 2023-12-25T21:09:26.256 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:26.254+0000 7f77c7e2a700 1 -- 172.21.15.174:0/4153437202 wait complete. 2023-12-25T21:09:26.256 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:26.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:26 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:24.590451+0000 mgr.smithi061.jxxeke (mgr.14164) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:26.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:26 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:26.253861+0000 mon.smithi061 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.174:0/4153437202' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:27.650 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:27.651 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:28.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:28 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:26.590780+0000 mgr.smithi061.jxxeke (mgr.14164) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:28.809 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.808+0000 7f18994b3700 1 -- 172.21.15.174:0/1614615483 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18940f5410 msgr2=0x7f18940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:28.810 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.808+0000 7f18994b3700 1 --2- 172.21.15.174:0/1614615483 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18940f5410 0x7f18940f5830 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f1884009fb0 tx=0x7f1884009b30).stop 2023-12-25T21:09:28.810 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.808+0000 7f18994b3700 1 -- 172.21.15.174:0/1614615483 shutdown_connections 2023-12-25T21:09:28.810 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.808+0000 7f18994b3700 1 --2- 172.21.15.174:0/1614615483 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18940f5410 0x7f18940f5830 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:28.810 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.808+0000 7f18994b3700 1 -- 172.21.15.174:0/1614615483 >> 172.21.15.174:0/1614615483 conn(0x7f18940f0b70 msgr2=0x7f18940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:28.810 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.808+0000 7f18994b3700 1 -- 172.21.15.174:0/1614615483 shutdown_connections 2023-12-25T21:09:28.810 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.808+0000 7f18994b3700 1 -- 172.21.15.174:0/1614615483 wait complete. 2023-12-25T21:09:28.812 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.809+0000 7f18994b3700 1 Processor -- start 2023-12-25T21:09:28.812 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.809+0000 7f18994b3700 1 -- start start 2023-12-25T21:09:28.812 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.809+0000 7f18994b3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18940f5410 0x7f18941088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:28.812 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.809+0000 7f18994b3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1894108dc0 con 0x7f18940f5410 2023-12-25T21:09:28.812 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.809+0000 7f1892ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18940f5410 0x7f18941088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:28.812 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.809+0000 7f1892ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18940f5410 0x7f18941088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:54032/0 (socket says 172.21.15.174:54032) 2023-12-25T21:09:28.812 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.809+0000 7f1892ffd700 1 -- 172.21.15.174:0/3644818711 learned_addr learned my addr 172.21.15.174:0/3644818711 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:28.812 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.810+0000 7f1892ffd700 1 -- 172.21.15.174:0/3644818711 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1884005040 con 0x7f18940f5410 2023-12-25T21:09:28.812 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.810+0000 7f1892ffd700 1 --2- 172.21.15.174:0/3644818711 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18940f5410 0x7f18941088d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f1884013510 tx=0x7f188400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:28.812 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.810+0000 7f187b7fe700 1 -- 172.21.15.174:0/3644818711 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1884009320 con 0x7f18940f5410 2023-12-25T21:09:28.812 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.810+0000 7f18994b3700 1 -- 172.21.15.174:0/3644818711 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1894108fc0 con 0x7f18940f5410 2023-12-25T21:09:28.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.810+0000 7f187b7fe700 1 -- 172.21.15.174:0/3644818711 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1884013f00 con 0x7f18940f5410 2023-12-25T21:09:28.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.810+0000 7f187b7fe700 1 -- 172.21.15.174:0/3644818711 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1884014d90 con 0x7f18940f5410 2023-12-25T21:09:28.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.810+0000 7f18994b3700 1 -- 172.21.15.174:0/3644818711 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1894109450 con 0x7f18940f5410 2023-12-25T21:09:28.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.811+0000 7f187b7fe700 1 -- 172.21.15.174:0/3644818711 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f18840263a0 con 0x7f18940f5410 2023-12-25T21:09:28.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.811+0000 7f187b7fe700 1 --2- 172.21.15.174:0/3644818711 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f187c02ff60 0x7f187c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:28.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.811+0000 7f187b7fe700 1 -- 172.21.15.174:0/3644818711 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f18840413e0 con 0x7f18940f5410 2023-12-25T21:09:28.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.811+0000 7f18994b3700 1 -- 172.21.15.174:0/3644818711 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1894059100 con 0x7f18940f5410 2023-12-25T21:09:28.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.811+0000 7f18927fc700 1 --2- 172.21.15.174:0/3644818711 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f187c02ff60 0x7f187c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:28.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.812+0000 7f18927fc700 1 --2- 172.21.15.174:0/3644818711 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f187c02ff60 0x7f187c032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f18880096a0 tx=0x7f1888006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:28.816 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:28.816+0000 7f187b7fe700 1 -- 172.21.15.174:0/3644818711 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1884012920 con 0x7f18940f5410 2023-12-25T21:09:29.027 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:29.026+0000 7f18994b3700 1 -- 172.21.15.174:0/3644818711 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18940fae60 con 0x7f18940f5410 2023-12-25T21:09:29.027 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:29.027+0000 7f187b7fe700 1 -- 172.21.15.174:0/3644818711 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f188402b030 con 0x7f18940f5410 2023-12-25T21:09:29.028 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:29.028 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:29.030 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:29.028+0000 7f18994b3700 1 -- 172.21.15.174:0/3644818711 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f187c02ff60 msgr2=0x7f187c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:29.030 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:29.028+0000 7f18994b3700 1 --2- 172.21.15.174:0/3644818711 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f187c02ff60 0x7f187c032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f18880096a0 tx=0x7f1888006b40).stop 2023-12-25T21:09:29.030 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:29.028+0000 7f18994b3700 1 -- 172.21.15.174:0/3644818711 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18940f5410 msgr2=0x7f18941088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:29.030 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:29.028+0000 7f18994b3700 1 --2- 172.21.15.174:0/3644818711 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18940f5410 0x7f18941088d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f1884013510 tx=0x7f188400b040).stop 2023-12-25T21:09:29.030 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:29.028+0000 7f18994b3700 1 -- 172.21.15.174:0/3644818711 shutdown_connections 2023-12-25T21:09:29.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:29.028+0000 7f18994b3700 1 --2- 172.21.15.174:0/3644818711 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f187c02ff60 0x7f187c032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:29.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:29.028+0000 7f18994b3700 1 --2- 172.21.15.174:0/3644818711 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18940f5410 0x7f18941088d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:29.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:29.028+0000 7f18994b3700 1 -- 172.21.15.174:0/3644818711 >> 172.21.15.174:0/3644818711 conn(0x7f18940f0b70 msgr2=0x7f189405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:29.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:29.028+0000 7f18994b3700 1 -- 172.21.15.174:0/3644818711 shutdown_connections 2023-12-25T21:09:29.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:29.028+0000 7f18994b3700 1 -- 172.21.15.174:0/3644818711 wait complete. 2023-12-25T21:09:29.031 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:29.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:29 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:29.028231+0000 mon.smithi061 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.174:0/3644818711' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:30.419 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:30.420 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:30.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:30 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:28.591137+0000 mgr.smithi061.jxxeke (mgr.14164) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:31.532 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.527+0000 7fba12153700 1 -- 172.21.15.174:0/4201030299 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba0c0f62a0 msgr2=0x7fba0c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:31.532 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.527+0000 7fba12153700 1 --2- 172.21.15.174:0/4201030299 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba0c0f62a0 0x7fba0c0f66c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fba08009fb0 tx=0x7fba08009b30).stop 2023-12-25T21:09:31.532 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.527+0000 7fba12153700 1 -- 172.21.15.174:0/4201030299 shutdown_connections 2023-12-25T21:09:31.532 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.527+0000 7fba12153700 1 --2- 172.21.15.174:0/4201030299 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba0c0f62a0 0x7fba0c0f66c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:31.532 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.527+0000 7fba12153700 1 -- 172.21.15.174:0/4201030299 >> 172.21.15.174:0/4201030299 conn(0x7fba0c0f1aa0 msgr2=0x7fba0c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:31.532 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.527+0000 7fba12153700 1 -- 172.21.15.174:0/4201030299 shutdown_connections 2023-12-25T21:09:31.532 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.527+0000 7fba12153700 1 -- 172.21.15.174:0/4201030299 wait complete. 2023-12-25T21:09:31.532 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.527+0000 7fba12153700 1 Processor -- start 2023-12-25T21:09:31.532 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.527+0000 7fba12153700 1 -- start start 2023-12-25T21:09:31.533 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.527+0000 7fba12153700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba0c0f62a0 0x7fba0c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:31.533 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.527+0000 7fba12153700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c1069a0 con 0x7fba0c0f62a0 2023-12-25T21:09:31.533 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.528+0000 7fba11151700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba0c0f62a0 0x7fba0c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:31.533 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.528+0000 7fba11151700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba0c0f62a0 0x7fba0c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:54058/0 (socket says 172.21.15.174:54058) 2023-12-25T21:09:31.533 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.528+0000 7fba11151700 1 -- 172.21.15.174:0/629023619 learned_addr learned my addr 172.21.15.174:0/629023619 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:31.533 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.528+0000 7fba11151700 1 -- 172.21.15.174:0/629023619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba08005040 con 0x7fba0c0f62a0 2023-12-25T21:09:31.533 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.528+0000 7fba11151700 1 --2- 172.21.15.174:0/629023619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba0c0f62a0 0x7fba0c1064b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fba08013510 tx=0x7fba0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:31.534 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.528+0000 7fba01ffb700 1 -- 172.21.15.174:0/629023619 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba08009320 con 0x7fba0c0f62a0 2023-12-25T21:09:31.534 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.528+0000 7fba12153700 1 -- 172.21.15.174:0/629023619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba0c106ba0 con 0x7fba0c0f62a0 2023-12-25T21:09:31.534 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.528+0000 7fba01ffb700 1 -- 172.21.15.174:0/629023619 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba08013f00 con 0x7fba0c0f62a0 2023-12-25T21:09:31.534 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.528+0000 7fba01ffb700 1 -- 172.21.15.174:0/629023619 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba08014dc0 con 0x7fba0c0f62a0 2023-12-25T21:09:31.534 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.529+0000 7fba12153700 1 -- 172.21.15.174:0/629023619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0c107030 con 0x7fba0c0f62a0 2023-12-25T21:09:31.534 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.529+0000 7fba01ffb700 1 -- 172.21.15.174:0/629023619 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba08014420 con 0x7fba0c0f62a0 2023-12-25T21:09:31.534 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.529+0000 7fba12153700 1 -- 172.21.15.174:0/629023619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba0c045ad0 con 0x7fba0c0f62a0 2023-12-25T21:09:31.534 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.529+0000 7fba01ffb700 1 --2- 172.21.15.174:0/629023619 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb9f802ff60 0x7fb9f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:31.534 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.530+0000 7fba01ffb700 1 -- 172.21.15.174:0/629023619 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba080410e0 con 0x7fba0c0f62a0 2023-12-25T21:09:31.534 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.530+0000 7fba10950700 1 --2- 172.21.15.174:0/629023619 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb9f802ff60 0x7fb9f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:31.534 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.530+0000 7fba10950700 1 --2- 172.21.15.174:0/629023619 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb9f802ff60 0x7fb9f8032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fb9fc0096a0 tx=0x7fb9fc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:31.535 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.534+0000 7fba01ffb700 1 -- 172.21.15.174:0/629023619 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba080095f0 con 0x7fba0c0f62a0 2023-12-25T21:09:31.740 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.740+0000 7fba12153700 1 -- 172.21.15.174:0/629023619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba0c107e80 con 0x7fba0c0f62a0 2023-12-25T21:09:31.741 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.740+0000 7fba01ffb700 1 -- 172.21.15.174:0/629023619 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fba0801cb10 con 0x7fba0c0f62a0 2023-12-25T21:09:31.741 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:31.741 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:31.743 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.742+0000 7fba12153700 1 -- 172.21.15.174:0/629023619 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb9f802ff60 msgr2=0x7fb9f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:31.743 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.742+0000 7fba12153700 1 --2- 172.21.15.174:0/629023619 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb9f802ff60 0x7fb9f8032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fb9fc0096a0 tx=0x7fb9fc006b40).stop 2023-12-25T21:09:31.743 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.742+0000 7fba12153700 1 -- 172.21.15.174:0/629023619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba0c0f62a0 msgr2=0x7fba0c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:31.743 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.742+0000 7fba12153700 1 --2- 172.21.15.174:0/629023619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba0c0f62a0 0x7fba0c1064b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fba08013510 tx=0x7fba0800b040).stop 2023-12-25T21:09:31.743 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.742+0000 7fba12153700 1 -- 172.21.15.174:0/629023619 shutdown_connections 2023-12-25T21:09:31.743 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.742+0000 7fba12153700 1 --2- 172.21.15.174:0/629023619 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb9f802ff60 0x7fb9f8032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:31.743 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.742+0000 7fba12153700 1 --2- 172.21.15.174:0/629023619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba0c0f62a0 0x7fba0c1064b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:31.743 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.742+0000 7fba12153700 1 -- 172.21.15.174:0/629023619 >> 172.21.15.174:0/629023619 conn(0x7fba0c0f1aa0 msgr2=0x7fba0c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:31.743 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.742+0000 7fba12153700 1 -- 172.21.15.174:0/629023619 shutdown_connections 2023-12-25T21:09:31.744 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:31.742+0000 7fba12153700 1 -- 172.21.15.174:0/629023619 wait complete. 2023-12-25T21:09:31.744 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:32.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:32 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:30.591511+0000 mgr.smithi061.jxxeke (mgr.14164) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:32.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:32 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:31.741663+0000 mon.smithi061 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.174:0/629023619' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:33.123 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:33.123 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:34.331 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.330+0000 7f8f20872700 1 -- 172.21.15.174:0/2764408769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f1c0f51a0 msgr2=0x7f8f1c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:34.331 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.330+0000 7f8f20872700 1 --2- 172.21.15.174:0/2764408769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f1c0f51a0 0x7f8f1c0f55c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f8f10009fb0 tx=0x7f8f10009b30).stop 2023-12-25T21:09:34.332 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.330+0000 7f8f20872700 1 -- 172.21.15.174:0/2764408769 shutdown_connections 2023-12-25T21:09:34.332 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.330+0000 7f8f20872700 1 --2- 172.21.15.174:0/2764408769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f1c0f51a0 0x7f8f1c0f55c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:34.332 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.330+0000 7f8f20872700 1 -- 172.21.15.174:0/2764408769 >> 172.21.15.174:0/2764408769 conn(0x7f8f1c0f09a0 msgr2=0x7f8f1c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:34.332 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.330+0000 7f8f20872700 1 -- 172.21.15.174:0/2764408769 shutdown_connections 2023-12-25T21:09:34.332 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.331+0000 7f8f20872700 1 -- 172.21.15.174:0/2764408769 wait complete. 2023-12-25T21:09:34.333 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.331+0000 7f8f20872700 1 Processor -- start 2023-12-25T21:09:34.333 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.331+0000 7f8f20872700 1 -- start start 2023-12-25T21:09:34.333 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.331+0000 7f8f20872700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f1c0f51a0 0x7f8f1c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:34.333 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.331+0000 7f8f20872700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c060240 con 0x7f8f1c0f51a0 2023-12-25T21:09:34.333 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.331+0000 7f8f1ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f1c0f51a0 0x7f8f1c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:34.333 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.332+0000 7f8f1ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f1c0f51a0 0x7f8f1c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:54070/0 (socket says 172.21.15.174:54070) 2023-12-25T21:09:34.334 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.332+0000 7f8f1ad9d700 1 -- 172.21.15.174:0/2769787757 learned_addr learned my addr 172.21.15.174:0/2769787757 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:34.334 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.332+0000 7f8f1ad9d700 1 -- 172.21.15.174:0/2769787757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f10005040 con 0x7f8f1c0f51a0 2023-12-25T21:09:34.334 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.332+0000 7f8f1ad9d700 1 --2- 172.21.15.174:0/2769787757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f1c0f51a0 0x7f8f1c1091b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f8f10013510 tx=0x7f8f1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:34.334 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.332+0000 7f8f037fe700 1 -- 172.21.15.174:0/2769787757 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f10009320 con 0x7f8f1c0f51a0 2023-12-25T21:09:34.334 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.332+0000 7f8f20872700 1 -- 172.21.15.174:0/2769787757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1c060440 con 0x7f8f1c0f51a0 2023-12-25T21:09:34.334 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.332+0000 7f8f037fe700 1 -- 172.21.15.174:0/2769787757 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f10013f00 con 0x7f8f1c0f51a0 2023-12-25T21:09:34.334 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.332+0000 7f8f037fe700 1 -- 172.21.15.174:0/2769787757 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f10014dc0 con 0x7f8f1c0f51a0 2023-12-25T21:09:34.334 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.332+0000 7f8f20872700 1 -- 172.21.15.174:0/2769787757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1c109860 con 0x7f8f1c0f51a0 2023-12-25T21:09:34.335 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.333+0000 7f8f037fe700 1 -- 172.21.15.174:0/2769787757 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f100095f0 con 0x7f8f1c0f51a0 2023-12-25T21:09:34.335 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.333+0000 7f8f037fe700 1 --2- 172.21.15.174:0/2769787757 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8f0402ff60 0x7f8f04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:34.335 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.333+0000 7f8f20872700 1 -- 172.21.15.174:0/2769787757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f1c045ad0 con 0x7f8f1c0f51a0 2023-12-25T21:09:34.335 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.333+0000 7f8f037fe700 1 -- 172.21.15.174:0/2769787757 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f10041100 con 0x7f8f1c0f51a0 2023-12-25T21:09:34.335 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.333+0000 7f8f1a59c700 1 --2- 172.21.15.174:0/2769787757 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8f0402ff60 0x7f8f04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:34.335 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.334+0000 7f8f1a59c700 1 --2- 172.21.15.174:0/2769787757 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8f0402ff60 0x7f8f04032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f8f0c0096a0 tx=0x7f8f0c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:34.339 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.338+0000 7f8f037fe700 1 -- 172.21.15.174:0/2769787757 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f1001c790 con 0x7f8f1c0f51a0 2023-12-25T21:09:34.544 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.543+0000 7f8f20872700 1 -- 172.21.15.174:0/2769787757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f1c10a080 con 0x7f8f1c0f51a0 2023-12-25T21:09:34.544 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.544+0000 7f8f037fe700 1 -- 172.21.15.174:0/2769787757 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8f10010230 con 0x7f8f1c0f51a0 2023-12-25T21:09:34.545 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:34.545 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:34.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.546+0000 7f8f20872700 1 -- 172.21.15.174:0/2769787757 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8f0402ff60 msgr2=0x7f8f04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:34.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.546+0000 7f8f20872700 1 --2- 172.21.15.174:0/2769787757 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8f0402ff60 0x7f8f04032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f8f0c0096a0 tx=0x7f8f0c006b40).stop 2023-12-25T21:09:34.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.546+0000 7f8f20872700 1 -- 172.21.15.174:0/2769787757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f1c0f51a0 msgr2=0x7f8f1c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:34.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.546+0000 7f8f20872700 1 --2- 172.21.15.174:0/2769787757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f1c0f51a0 0x7f8f1c1091b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f8f10013510 tx=0x7f8f1000b040).stop 2023-12-25T21:09:34.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.546+0000 7f8f20872700 1 -- 172.21.15.174:0/2769787757 shutdown_connections 2023-12-25T21:09:34.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.546+0000 7f8f20872700 1 --2- 172.21.15.174:0/2769787757 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8f0402ff60 0x7f8f04032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:34.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.546+0000 7f8f20872700 1 --2- 172.21.15.174:0/2769787757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f1c0f51a0 0x7f8f1c1091b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:34.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.546+0000 7f8f20872700 1 -- 172.21.15.174:0/2769787757 >> 172.21.15.174:0/2769787757 conn(0x7f8f1c0f09a0 msgr2=0x7f8f1c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:34.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.546+0000 7f8f20872700 1 -- 172.21.15.174:0/2769787757 shutdown_connections 2023-12-25T21:09:34.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:34.546+0000 7f8f20872700 1 -- 172.21.15.174:0/2769787757 wait complete. 2023-12-25T21:09:34.547 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:34.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:34 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:32.591803+0000 mgr.smithi061.jxxeke (mgr.14164) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:35.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:35 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:34.545474+0000 mon.smithi061 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.174:0/2769787757' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:35.947 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:35.947 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:36.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:36 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:34.592179+0000 mgr.smithi061.jxxeke (mgr.14164) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:37.085 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.083+0000 7f2b8dcdd700 1 -- 172.21.15.174:0/796444740 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b880f5410 msgr2=0x7f2b880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:37.085 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.083+0000 7f2b8dcdd700 1 --2- 172.21.15.174:0/796444740 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b880f5410 0x7f2b880f5830 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f2b7c009fb0 tx=0x7f2b7c009b30).stop 2023-12-25T21:09:37.085 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.083+0000 7f2b8dcdd700 1 -- 172.21.15.174:0/796444740 shutdown_connections 2023-12-25T21:09:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.083+0000 7f2b8dcdd700 1 --2- 172.21.15.174:0/796444740 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b880f5410 0x7f2b880f5830 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.083+0000 7f2b8dcdd700 1 -- 172.21.15.174:0/796444740 >> 172.21.15.174:0/796444740 conn(0x7f2b880f0b70 msgr2=0x7f2b880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.083+0000 7f2b8dcdd700 1 -- 172.21.15.174:0/796444740 shutdown_connections 2023-12-25T21:09:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.083+0000 7f2b8dcdd700 1 -- 172.21.15.174:0/796444740 wait complete. 2023-12-25T21:09:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.084+0000 7f2b8dcdd700 1 Processor -- start 2023-12-25T21:09:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.084+0000 7f2b8dcdd700 1 -- start start 2023-12-25T21:09:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.084+0000 7f2b8dcdd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b880f5410 0x7f2b88109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.084+0000 7f2b8dcdd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b88060220 con 0x7f2b880f5410 2023-12-25T21:09:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.084+0000 7f2b877fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b880f5410 0x7f2b88109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.084+0000 7f2b877fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b880f5410 0x7f2b88109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:58220/0 (socket says 172.21.15.174:58220) 2023-12-25T21:09:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.085+0000 7f2b877fe700 1 -- 172.21.15.174:0/1717482173 learned_addr learned my addr 172.21.15.174:0/1717482173 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.085+0000 7f2b877fe700 1 -- 172.21.15.174:0/1717482173 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7c005040 con 0x7f2b880f5410 2023-12-25T21:09:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.085+0000 7f2b877fe700 1 --2- 172.21.15.174:0/1717482173 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b880f5410 0x7f2b88109450 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f2b7c013510 tx=0x7f2b7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.085+0000 7f2b6ffff700 1 -- 172.21.15.174:0/1717482173 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b7c009320 con 0x7f2b880f5410 2023-12-25T21:09:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.085+0000 7f2b6ffff700 1 -- 172.21.15.174:0/1717482173 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b7c013f00 con 0x7f2b880f5410 2023-12-25T21:09:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.085+0000 7f2b8dcdd700 1 -- 172.21.15.174:0/1717482173 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b88060420 con 0x7f2b880f5410 2023-12-25T21:09:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.085+0000 7f2b6ffff700 1 -- 172.21.15.174:0/1717482173 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b7c014d90 con 0x7f2b880f5410 2023-12-25T21:09:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.085+0000 7f2b8dcdd700 1 -- 172.21.15.174:0/1717482173 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b88109b00 con 0x7f2b880f5410 2023-12-25T21:09:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.086+0000 7f2b6ffff700 1 -- 172.21.15.174:0/1717482173 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2b7c0095f0 con 0x7f2b880f5410 2023-12-25T21:09:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.086+0000 7f2b8dcdd700 1 -- 172.21.15.174:0/1717482173 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b88059100 con 0x7f2b880f5410 2023-12-25T21:09:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.086+0000 7f2b6ffff700 1 --2- 172.21.15.174:0/1717482173 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2b7002ff60 0x7f2b70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.086+0000 7f2b6ffff700 1 -- 172.21.15.174:0/1717482173 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b7c0410c0 con 0x7f2b880f5410 2023-12-25T21:09:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.086+0000 7f2b86ffd700 1 --2- 172.21.15.174:0/1717482173 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2b7002ff60 0x7f2b70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.087+0000 7f2b86ffd700 1 --2- 172.21.15.174:0/1717482173 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2b7002ff60 0x7f2b70032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f2b780096a0 tx=0x7f2b78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:37.091 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.091+0000 7f2b6ffff700 1 -- 172.21.15.174:0/1717482173 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b7c014420 con 0x7f2b880f5410 2023-12-25T21:09:37.296 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.296+0000 7f2b8dcdd700 1 -- 172.21.15.174:0/1717482173 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b88045ad0 con 0x7f2b880f5410 2023-12-25T21:09:37.297 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.296+0000 7f2b6ffff700 1 -- 172.21.15.174:0/1717482173 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2b7c01d7f0 con 0x7f2b880f5410 2023-12-25T21:09:37.297 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:37.297 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:37.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.298+0000 7f2b8dcdd700 1 -- 172.21.15.174:0/1717482173 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2b7002ff60 msgr2=0x7f2b70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:37.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.298+0000 7f2b8dcdd700 1 --2- 172.21.15.174:0/1717482173 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2b7002ff60 0x7f2b70032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f2b780096a0 tx=0x7f2b78006b40).stop 2023-12-25T21:09:37.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.298+0000 7f2b8dcdd700 1 -- 172.21.15.174:0/1717482173 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b880f5410 msgr2=0x7f2b88109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:37.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.298+0000 7f2b8dcdd700 1 --2- 172.21.15.174:0/1717482173 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b880f5410 0x7f2b88109450 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f2b7c013510 tx=0x7f2b7c00b040).stop 2023-12-25T21:09:37.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.298+0000 7f2b8dcdd700 1 -- 172.21.15.174:0/1717482173 shutdown_connections 2023-12-25T21:09:37.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.298+0000 7f2b8dcdd700 1 --2- 172.21.15.174:0/1717482173 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2b7002ff60 0x7f2b70032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:37.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.298+0000 7f2b8dcdd700 1 --2- 172.21.15.174:0/1717482173 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b880f5410 0x7f2b88109450 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:37.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.298+0000 7f2b8dcdd700 1 -- 172.21.15.174:0/1717482173 >> 172.21.15.174:0/1717482173 conn(0x7f2b880f0b70 msgr2=0x7f2b880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:37.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.298+0000 7f2b8dcdd700 1 -- 172.21.15.174:0/1717482173 shutdown_connections 2023-12-25T21:09:37.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:37.298+0000 7f2b8dcdd700 1 -- 172.21.15.174:0/1717482173 wait complete. 2023-12-25T21:09:37.300 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:37.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:37 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:37.297648+0000 mon.smithi061 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.174:0/1717482173' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:38.691 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:38.691 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:38.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:38 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:36.592486+0000 mgr.smithi061.jxxeke (mgr.14164) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:39.901 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.899+0000 7fe73ad23700 1 -- 172.21.15.174:0/2002514825 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 msgr2=0x7fe7340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:39.901 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.899+0000 7fe73ad23700 1 --2- 172.21.15.174:0/2002514825 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 0x7fe7340f5830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fe728009fb0 tx=0x7fe728009b30).stop 2023-12-25T21:09:39.901 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.899+0000 7fe73ad23700 1 -- 172.21.15.174:0/2002514825 shutdown_connections 2023-12-25T21:09:39.901 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.899+0000 7fe73ad23700 1 --2- 172.21.15.174:0/2002514825 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 0x7fe7340f5830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:39.901 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.899+0000 7fe73ad23700 1 -- 172.21.15.174:0/2002514825 >> 172.21.15.174:0/2002514825 conn(0x7fe7340f0b70 msgr2=0x7fe7340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:39.901 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.899+0000 7fe73ad23700 1 -- 172.21.15.174:0/2002514825 shutdown_connections 2023-12-25T21:09:39.901 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.899+0000 7fe73ad23700 1 -- 172.21.15.174:0/2002514825 wait complete. 2023-12-25T21:09:39.902 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.899+0000 7fe73ad23700 1 Processor -- start 2023-12-25T21:09:39.902 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.899+0000 7fe73ad23700 1 -- start start 2023-12-25T21:09:39.902 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.900+0000 7fe73ad23700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 0x7fe7340fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:39.902 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.900+0000 7fe73ad23700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7340fb820 con 0x7fe7340f5410 2023-12-25T21:09:39.902 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.900+0000 7fe738abf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 0x7fe7340fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:39.902 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.900+0000 7fe738abf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 0x7fe7340fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:58248/0 (socket says 172.21.15.174:58248) 2023-12-25T21:09:39.902 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.900+0000 7fe738abf700 1 -- 172.21.15.174:0/4270061556 learned_addr learned my addr 172.21.15.174:0/4270061556 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:39.902 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.900+0000 7fe738abf700 1 -- 172.21.15.174:0/4270061556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe728005040 con 0x7fe7340f5410 2023-12-25T21:09:39.902 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.900+0000 7fe738abf700 1 --2- 172.21.15.174:0/4270061556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 0x7fe7340fd3a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe728013510 tx=0x7fe72800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:39.903 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.901+0000 7fe7317fa700 1 -- 172.21.15.174:0/4270061556 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe728009320 con 0x7fe7340f5410 2023-12-25T21:09:39.903 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.901+0000 7fe73ad23700 1 -- 172.21.15.174:0/4270061556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7340fb9c0 con 0x7fe7340f5410 2023-12-25T21:09:39.904 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.901+0000 7fe7317fa700 1 -- 172.21.15.174:0/4270061556 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe728013f00 con 0x7fe7340f5410 2023-12-25T21:09:39.904 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.901+0000 7fe7317fa700 1 -- 172.21.15.174:0/4270061556 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe728014d90 con 0x7fe7340f5410 2023-12-25T21:09:39.904 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.901+0000 7fe73ad23700 1 -- 172.21.15.174:0/4270061556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7340fbe50 con 0x7fe7340f5410 2023-12-25T21:09:39.904 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.901+0000 7fe7317fa700 1 -- 172.21.15.174:0/4270061556 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7280095f0 con 0x7fe7340f5410 2023-12-25T21:09:39.904 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.901+0000 7fe73ad23700 1 -- 172.21.15.174:0/4270061556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe734059100 con 0x7fe7340f5410 2023-12-25T21:09:39.904 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.902+0000 7fe7317fa700 1 --2- 172.21.15.174:0/4270061556 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe72402ff60 0x7fe724032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:39.904 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.902+0000 7fe7317fa700 1 -- 172.21.15.174:0/4270061556 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe7280410c0 con 0x7fe7340f5410 2023-12-25T21:09:39.904 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.902+0000 7fe733fff700 1 --2- 172.21.15.174:0/4270061556 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe72402ff60 0x7fe724032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:39.904 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.902+0000 7fe733fff700 1 --2- 172.21.15.174:0/4270061556 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe72402ff60 0x7fe724032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fe7200096a0 tx=0x7fe720006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:39.907 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:39.906+0000 7fe7317fa700 1 -- 172.21.15.174:0/4270061556 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe728014420 con 0x7fe7340f5410 2023-12-25T21:09:40.111 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:40.111+0000 7fe73ad23700 1 -- 172.21.15.174:0/4270061556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7340fae60 con 0x7fe7340f5410 2023-12-25T21:09:40.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:40.111+0000 7fe7317fa700 1 -- 172.21.15.174:0/4270061556 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe728010230 con 0x7fe7340f5410 2023-12-25T21:09:40.112 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:40.113 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:40.114 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:40.113+0000 7fe73ad23700 1 -- 172.21.15.174:0/4270061556 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe72402ff60 msgr2=0x7fe724032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:40.114 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:40.113+0000 7fe73ad23700 1 --2- 172.21.15.174:0/4270061556 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe72402ff60 0x7fe724032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fe7200096a0 tx=0x7fe720006b40).stop 2023-12-25T21:09:40.114 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:40.113+0000 7fe73ad23700 1 -- 172.21.15.174:0/4270061556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 msgr2=0x7fe7340fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:40.114 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:40.113+0000 7fe73ad23700 1 --2- 172.21.15.174:0/4270061556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 0x7fe7340fd3a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe728013510 tx=0x7fe72800b040).stop 2023-12-25T21:09:40.114 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:40.113+0000 7fe73ad23700 1 -- 172.21.15.174:0/4270061556 shutdown_connections 2023-12-25T21:09:40.114 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:40.113+0000 7fe73ad23700 1 --2- 172.21.15.174:0/4270061556 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe72402ff60 0x7fe724032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:40.114 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:40.113+0000 7fe73ad23700 1 --2- 172.21.15.174:0/4270061556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7340f5410 0x7fe7340fd3a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:40.114 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:40.113+0000 7fe73ad23700 1 -- 172.21.15.174:0/4270061556 >> 172.21.15.174:0/4270061556 conn(0x7fe7340f0b70 msgr2=0x7fe73405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:40.114 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:40.113+0000 7fe73ad23700 1 -- 172.21.15.174:0/4270061556 shutdown_connections 2023-12-25T21:09:40.114 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:40.113+0000 7fe73ad23700 1 -- 172.21.15.174:0/4270061556 wait complete. 2023-12-25T21:09:40.115 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:40.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:40 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:38.592811+0000 mgr.smithi061.jxxeke (mgr.14164) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:40.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:40 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:40.112702+0000 mon.smithi061 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.174:0/4270061556' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:41.505 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:41.505 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:42.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:42 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:40.593131+0000 mgr.smithi061.jxxeke (mgr.14164) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:42.726 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.725+0000 7f6006670700 1 -- 172.21.15.174:0/1440423635 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6000060280 msgr2=0x7f60000f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:42.726 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.725+0000 7f6006670700 1 --2- 172.21.15.174:0/1440423635 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6000060280 0x7f60000f90f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f5fe8009fb0 tx=0x7f5fe8009b30).stop 2023-12-25T21:09:42.726 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.725+0000 7f6006670700 1 -- 172.21.15.174:0/1440423635 shutdown_connections 2023-12-25T21:09:42.726 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.725+0000 7f6006670700 1 --2- 172.21.15.174:0/1440423635 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6000060280 0x7f60000f90f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:42.726 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.725+0000 7f6006670700 1 -- 172.21.15.174:0/1440423635 >> 172.21.15.174:0/1440423635 conn(0x7f60000f0b70 msgr2=0x7f60000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:42.726 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.725+0000 7f6006670700 1 -- 172.21.15.174:0/1440423635 shutdown_connections 2023-12-25T21:09:42.727 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.725+0000 7f6006670700 1 -- 172.21.15.174:0/1440423635 wait complete. 2023-12-25T21:09:42.727 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.725+0000 7f6006670700 1 Processor -- start 2023-12-25T21:09:42.727 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.725+0000 7f6006670700 1 -- start start 2023-12-25T21:09:42.727 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.726+0000 7f6006670700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6000060280 0x7f60001066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:42.727 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.726+0000 7f6006670700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6000106bd0 con 0x7f6000060280 2023-12-25T21:09:42.728 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.726+0000 7f5ffffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6000060280 0x7f60001066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:42.728 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.726+0000 7f5ffffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6000060280 0x7f60001066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:58262/0 (socket says 172.21.15.174:58262) 2023-12-25T21:09:42.728 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.726+0000 7f5ffffff700 1 -- 172.21.15.174:0/1046761070 learned_addr learned my addr 172.21.15.174:0/1046761070 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:42.728 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.726+0000 7f5ffffff700 1 -- 172.21.15.174:0/1046761070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe8005040 con 0x7f6000060280 2023-12-25T21:09:42.728 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.726+0000 7f5ffffff700 1 --2- 172.21.15.174:0/1046761070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6000060280 0x7f60001066e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f5fe8013510 tx=0x7f5fe800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:42.729 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.726+0000 7f5ffcff9700 1 -- 172.21.15.174:0/1046761070 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fe8009320 con 0x7f6000060280 2023-12-25T21:09:42.729 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.726+0000 7f6006670700 1 -- 172.21.15.174:0/1046761070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6000106dd0 con 0x7f6000060280 2023-12-25T21:09:42.729 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.726+0000 7f5ffcff9700 1 -- 172.21.15.174:0/1046761070 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5fe8013f00 con 0x7f6000060280 2023-12-25T21:09:42.729 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.726+0000 7f5ffcff9700 1 -- 172.21.15.174:0/1046761070 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fe8014dc0 con 0x7f6000060280 2023-12-25T21:09:42.729 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.726+0000 7f6006670700 1 -- 172.21.15.174:0/1046761070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6000107260 con 0x7f6000060280 2023-12-25T21:09:42.729 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.727+0000 7f5ffcff9700 1 -- 172.21.15.174:0/1046761070 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5fe80095f0 con 0x7f6000060280 2023-12-25T21:09:42.729 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.727+0000 7f6006670700 1 -- 172.21.15.174:0/1046761070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6000045ad0 con 0x7f6000060280 2023-12-25T21:09:42.729 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.727+0000 7f5ffcff9700 1 --2- 172.21.15.174:0/1046761070 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5fec02ff60 0x7f5fec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:42.729 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.727+0000 7f5ffcff9700 1 -- 172.21.15.174:0/1046761070 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5fe8041000 con 0x7f6000060280 2023-12-25T21:09:42.729 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.728+0000 7f5fff7fe700 1 --2- 172.21.15.174:0/1046761070 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5fec02ff60 0x7f5fec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:42.730 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.728+0000 7f5fff7fe700 1 --2- 172.21.15.174:0/1046761070 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5fec02ff60 0x7f5fec032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f5ff00096a0 tx=0x7f5ff0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:42.733 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.732+0000 7f5ffcff9700 1 -- 172.21.15.174:0/1046761070 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5fe801c790 con 0x7f6000060280 2023-12-25T21:09:42.937 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.936+0000 7f6006670700 1 -- 172.21.15.174:0/1046761070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6000108110 con 0x7f6000060280 2023-12-25T21:09:42.937 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.937+0000 7f5ffcff9700 1 -- 172.21.15.174:0/1046761070 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5fe8010230 con 0x7f6000060280 2023-12-25T21:09:42.938 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:42.938 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:42.939 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.938+0000 7f6006670700 1 -- 172.21.15.174:0/1046761070 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5fec02ff60 msgr2=0x7f5fec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:42.939 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.938+0000 7f6006670700 1 --2- 172.21.15.174:0/1046761070 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5fec02ff60 0x7f5fec032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f5ff00096a0 tx=0x7f5ff0006b40).stop 2023-12-25T21:09:42.939 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.938+0000 7f6006670700 1 -- 172.21.15.174:0/1046761070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6000060280 msgr2=0x7f60001066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:42.939 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.938+0000 7f6006670700 1 --2- 172.21.15.174:0/1046761070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6000060280 0x7f60001066e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f5fe8013510 tx=0x7f5fe800b040).stop 2023-12-25T21:09:42.939 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.938+0000 7f6006670700 1 -- 172.21.15.174:0/1046761070 shutdown_connections 2023-12-25T21:09:42.939 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.938+0000 7f6006670700 1 --2- 172.21.15.174:0/1046761070 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5fec02ff60 0x7f5fec032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:42.939 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.938+0000 7f6006670700 1 --2- 172.21.15.174:0/1046761070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6000060280 0x7f60001066e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:42.939 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.938+0000 7f6006670700 1 -- 172.21.15.174:0/1046761070 >> 172.21.15.174:0/1046761070 conn(0x7f60000f0b70 msgr2=0x7f60000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:42.939 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.938+0000 7f6006670700 1 -- 172.21.15.174:0/1046761070 shutdown_connections 2023-12-25T21:09:42.940 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:42.938+0000 7f6006670700 1 -- 172.21.15.174:0/1046761070 wait complete. 2023-12-25T21:09:42.940 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:43.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:43 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:42.938136+0000 mon.smithi061 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.174:0/1046761070' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:44.274 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:44.275 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:44.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:44 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:42.593401+0000 mgr.smithi061.jxxeke (mgr.14164) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:45.440 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.437+0000 7f6a8d12c700 1 -- 172.21.15.174:0/21031475 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a88060280 msgr2=0x7f6a880f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:45.440 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.437+0000 7f6a8d12c700 1 --2- 172.21.15.174:0/21031475 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a88060280 0x7f6a880f90f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f6a7c009fb0 tx=0x7f6a7c009b30).stop 2023-12-25T21:09:45.440 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.437+0000 7f6a8d12c700 1 -- 172.21.15.174:0/21031475 shutdown_connections 2023-12-25T21:09:45.440 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.437+0000 7f6a8d12c700 1 --2- 172.21.15.174:0/21031475 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a88060280 0x7f6a880f90f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:45.440 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.437+0000 7f6a8d12c700 1 -- 172.21.15.174:0/21031475 >> 172.21.15.174:0/21031475 conn(0x7f6a880f0b70 msgr2=0x7f6a880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:45.440 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.437+0000 7f6a8d12c700 1 -- 172.21.15.174:0/21031475 shutdown_connections 2023-12-25T21:09:45.440 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.437+0000 7f6a8d12c700 1 -- 172.21.15.174:0/21031475 wait complete. 2023-12-25T21:09:45.440 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.438+0000 7f6a8d12c700 1 Processor -- start 2023-12-25T21:09:45.441 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.438+0000 7f6a8d12c700 1 -- start start 2023-12-25T21:09:45.441 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.438+0000 7f6a8d12c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a88060280 0x7f6a881066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:45.441 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.438+0000 7f6a8d12c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a88106bd0 con 0x7f6a88060280 2023-12-25T21:09:45.441 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.438+0000 7f6a86d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a88060280 0x7f6a881066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:45.441 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.438+0000 7f6a86d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a88060280 0x7f6a881066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:58280/0 (socket says 172.21.15.174:58280) 2023-12-25T21:09:45.441 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.438+0000 7f6a86d9d700 1 -- 172.21.15.174:0/1381183944 learned_addr learned my addr 172.21.15.174:0/1381183944 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:45.441 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.439+0000 7f6a86d9d700 1 -- 172.21.15.174:0/1381183944 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7c005040 con 0x7f6a88060280 2023-12-25T21:09:45.441 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.439+0000 7f6a86d9d700 1 --2- 172.21.15.174:0/1381183944 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a88060280 0x7f6a881066e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f6a7c013510 tx=0x7f6a7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:45.441 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.439+0000 7f6a6f7fe700 1 -- 172.21.15.174:0/1381183944 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a7c009320 con 0x7f6a88060280 2023-12-25T21:09:45.441 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.439+0000 7f6a6f7fe700 1 -- 172.21.15.174:0/1381183944 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a7c013f00 con 0x7f6a88060280 2023-12-25T21:09:45.441 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.439+0000 7f6a8d12c700 1 -- 172.21.15.174:0/1381183944 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a88106dd0 con 0x7f6a88060280 2023-12-25T21:09:45.441 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.439+0000 7f6a6f7fe700 1 -- 172.21.15.174:0/1381183944 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a7c014dc0 con 0x7f6a88060280 2023-12-25T21:09:45.441 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.439+0000 7f6a8d12c700 1 -- 172.21.15.174:0/1381183944 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a88107260 con 0x7f6a88060280 2023-12-25T21:09:45.442 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.440+0000 7f6a6f7fe700 1 -- 172.21.15.174:0/1381183944 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6a7c014420 con 0x7f6a88060280 2023-12-25T21:09:45.442 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.440+0000 7f6a8d12c700 1 -- 172.21.15.174:0/1381183944 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a88045ad0 con 0x7f6a88060280 2023-12-25T21:09:45.442 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.440+0000 7f6a6f7fe700 1 --2- 172.21.15.174:0/1381183944 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6a7002ff60 0x7f6a70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:45.443 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.440+0000 7f6a6f7fe700 1 -- 172.21.15.174:0/1381183944 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6a7c0410e0 con 0x7f6a88060280 2023-12-25T21:09:45.443 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.440+0000 7f6a8659c700 1 --2- 172.21.15.174:0/1381183944 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6a7002ff60 0x7f6a70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:45.443 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.441+0000 7f6a8659c700 1 --2- 172.21.15.174:0/1381183944 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6a7002ff60 0x7f6a70032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f6a780096a0 tx=0x7f6a78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:45.445 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.445+0000 7f6a6f7fe700 1 -- 172.21.15.174:0/1381183944 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a7c0095f0 con 0x7f6a88060280 2023-12-25T21:09:45.653 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.652+0000 7f6a8d12c700 1 -- 172.21.15.174:0/1381183944 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a88108110 con 0x7f6a88060280 2023-12-25T21:09:45.653 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.652+0000 7f6a6f7fe700 1 -- 172.21.15.174:0/1381183944 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6a7c01cb10 con 0x7f6a88060280 2023-12-25T21:09:45.653 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:45.653 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:45.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.654+0000 7f6a8d12c700 1 -- 172.21.15.174:0/1381183944 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6a7002ff60 msgr2=0x7f6a70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:45.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.654+0000 7f6a8d12c700 1 --2- 172.21.15.174:0/1381183944 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6a7002ff60 0x7f6a70032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f6a780096a0 tx=0x7f6a78006b40).stop 2023-12-25T21:09:45.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.654+0000 7f6a8d12c700 1 -- 172.21.15.174:0/1381183944 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a88060280 msgr2=0x7f6a881066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:45.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.654+0000 7f6a8d12c700 1 --2- 172.21.15.174:0/1381183944 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a88060280 0x7f6a881066e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f6a7c013510 tx=0x7f6a7c00b040).stop 2023-12-25T21:09:45.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.654+0000 7f6a8d12c700 1 -- 172.21.15.174:0/1381183944 shutdown_connections 2023-12-25T21:09:45.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.654+0000 7f6a8d12c700 1 --2- 172.21.15.174:0/1381183944 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6a7002ff60 0x7f6a70032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:45.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.654+0000 7f6a8d12c700 1 --2- 172.21.15.174:0/1381183944 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a88060280 0x7f6a881066e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:45.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.654+0000 7f6a8d12c700 1 -- 172.21.15.174:0/1381183944 >> 172.21.15.174:0/1381183944 conn(0x7f6a880f0b70 msgr2=0x7f6a880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:45.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.654+0000 7f6a8d12c700 1 -- 172.21.15.174:0/1381183944 shutdown_connections 2023-12-25T21:09:45.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:45.654+0000 7f6a8d12c700 1 -- 172.21.15.174:0/1381183944 wait complete. 2023-12-25T21:09:45.656 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:46.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:46 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:44.593642+0000 mgr.smithi061.jxxeke (mgr.14164) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:46.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:46 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:45.653901+0000 mon.smithi061 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.174:0/1381183944' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:47.061 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:47.062 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:48.207 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.205+0000 7faf8c610700 1 -- 172.21.15.174:0/3292934426 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf840f5410 msgr2=0x7faf840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:48.207 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.205+0000 7faf8c610700 1 --2- 172.21.15.174:0/3292934426 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf840f5410 0x7faf840f5830 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7faf74009fb0 tx=0x7faf74009b30).stop 2023-12-25T21:09:48.207 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.205+0000 7faf8c610700 1 -- 172.21.15.174:0/3292934426 shutdown_connections 2023-12-25T21:09:48.207 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.205+0000 7faf8c610700 1 --2- 172.21.15.174:0/3292934426 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf840f5410 0x7faf840f5830 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:48.207 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.205+0000 7faf8c610700 1 -- 172.21.15.174:0/3292934426 >> 172.21.15.174:0/3292934426 conn(0x7faf840f0b70 msgr2=0x7faf840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:48.208 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.205+0000 7faf8c610700 1 -- 172.21.15.174:0/3292934426 shutdown_connections 2023-12-25T21:09:48.208 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.205+0000 7faf8c610700 1 -- 172.21.15.174:0/3292934426 wait complete. 2023-12-25T21:09:48.208 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.206+0000 7faf8c610700 1 Processor -- start 2023-12-25T21:09:48.208 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.206+0000 7faf8c610700 1 -- start start 2023-12-25T21:09:48.208 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.206+0000 7faf8c610700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf840f5410 0x7faf841093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:48.208 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.206+0000 7faf8c610700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf84060280 con 0x7faf840f5410 2023-12-25T21:09:48.208 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.206+0000 7faf8a3ac700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf840f5410 0x7faf841093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:48.208 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.206+0000 7faf8a3ac700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf840f5410 0x7faf841093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:37284/0 (socket says 172.21.15.174:37284) 2023-12-25T21:09:48.208 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.206+0000 7faf8a3ac700 1 -- 172.21.15.174:0/4037278336 learned_addr learned my addr 172.21.15.174:0/4037278336 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:48.208 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.206+0000 7faf8a3ac700 1 -- 172.21.15.174:0/4037278336 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf74005040 con 0x7faf840f5410 2023-12-25T21:09:48.208 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.207+0000 7faf8a3ac700 1 --2- 172.21.15.174:0/4037278336 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf840f5410 0x7faf841093f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7faf74013510 tx=0x7faf7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:48.209 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.207+0000 7faf7affd700 1 -- 172.21.15.174:0/4037278336 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf74009320 con 0x7faf840f5410 2023-12-25T21:09:48.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.207+0000 7faf8c610700 1 -- 172.21.15.174:0/4037278336 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf84060480 con 0x7faf840f5410 2023-12-25T21:09:48.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.207+0000 7faf7affd700 1 -- 172.21.15.174:0/4037278336 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf74013f00 con 0x7faf840f5410 2023-12-25T21:09:48.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.207+0000 7faf7affd700 1 -- 172.21.15.174:0/4037278336 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf74014d90 con 0x7faf840f5410 2023-12-25T21:09:48.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.207+0000 7faf8c610700 1 -- 172.21.15.174:0/4037278336 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf84109b20 con 0x7faf840f5410 2023-12-25T21:09:48.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.208+0000 7faf7affd700 1 -- 172.21.15.174:0/4037278336 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf740263a0 con 0x7faf840f5410 2023-12-25T21:09:48.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.208+0000 7faf7affd700 1 --2- 172.21.15.174:0/4037278336 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7faf7002ff60 0x7faf70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:48.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.208+0000 7faf7affd700 1 -- 172.21.15.174:0/4037278336 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf74041030 con 0x7faf840f5410 2023-12-25T21:09:48.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.208+0000 7faf8c610700 1 -- 172.21.15.174:0/4037278336 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf84045ad0 con 0x7faf840f5410 2023-12-25T21:09:48.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.208+0000 7faf89bab700 1 --2- 172.21.15.174:0/4037278336 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7faf7002ff60 0x7faf70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:48.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.208+0000 7faf89bab700 1 --2- 172.21.15.174:0/4037278336 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7faf7002ff60 0x7faf70032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7faf800096a0 tx=0x7faf80006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:48.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.213+0000 7faf7affd700 1 -- 172.21.15.174:0/4037278336 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf74012920 con 0x7faf840f5410 2023-12-25T21:09:48.420 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.419+0000 7faf8c610700 1 -- 172.21.15.174:0/4037278336 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf8410a240 con 0x7faf840f5410 2023-12-25T21:09:48.421 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.420+0000 7faf7affd700 1 -- 172.21.15.174:0/4037278336 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faf7402b020 con 0x7faf840f5410 2023-12-25T21:09:48.421 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:48.421 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:48.423 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.422+0000 7faf8c610700 1 -- 172.21.15.174:0/4037278336 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7faf7002ff60 msgr2=0x7faf70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:48.423 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.422+0000 7faf8c610700 1 --2- 172.21.15.174:0/4037278336 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7faf7002ff60 0x7faf70032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7faf800096a0 tx=0x7faf80006b40).stop 2023-12-25T21:09:48.423 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.422+0000 7faf8c610700 1 -- 172.21.15.174:0/4037278336 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf840f5410 msgr2=0x7faf841093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:48.423 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.422+0000 7faf8c610700 1 --2- 172.21.15.174:0/4037278336 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf840f5410 0x7faf841093f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7faf74013510 tx=0x7faf7400b040).stop 2023-12-25T21:09:48.423 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.422+0000 7faf8c610700 1 -- 172.21.15.174:0/4037278336 shutdown_connections 2023-12-25T21:09:48.423 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.422+0000 7faf8c610700 1 --2- 172.21.15.174:0/4037278336 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7faf7002ff60 0x7faf70032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:48.423 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.422+0000 7faf8c610700 1 --2- 172.21.15.174:0/4037278336 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf840f5410 0x7faf841093f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:48.423 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.422+0000 7faf8c610700 1 -- 172.21.15.174:0/4037278336 >> 172.21.15.174:0/4037278336 conn(0x7faf840f0b70 msgr2=0x7faf840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:48.423 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.422+0000 7faf8c610700 1 -- 172.21.15.174:0/4037278336 shutdown_connections 2023-12-25T21:09:48.423 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:48.422+0000 7faf8c610700 1 -- 172.21.15.174:0/4037278336 wait complete. 2023-12-25T21:09:48.424 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:48.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:48 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:46.593955+0000 mgr.smithi061.jxxeke (mgr.14164) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:49.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:49 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:48.421665+0000 mon.smithi061 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.174:0/4037278336' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:49.829 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:49.830 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:50.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:50 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:48.594261+0000 mgr.smithi061.jxxeke (mgr.14164) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:50.975 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.973+0000 7fbd90341700 1 -- 172.21.15.174:0/3653354361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd88060280 msgr2=0x7fbd880f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:50.975 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.973+0000 7fbd90341700 1 --2- 172.21.15.174:0/3653354361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd88060280 0x7fbd880f90f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fbd78009fb0 tx=0x7fbd78009b30).stop 2023-12-25T21:09:50.975 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.973+0000 7fbd90341700 1 -- 172.21.15.174:0/3653354361 shutdown_connections 2023-12-25T21:09:50.975 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.973+0000 7fbd90341700 1 --2- 172.21.15.174:0/3653354361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd88060280 0x7fbd880f90f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:50.975 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.973+0000 7fbd90341700 1 -- 172.21.15.174:0/3653354361 >> 172.21.15.174:0/3653354361 conn(0x7fbd880f0b70 msgr2=0x7fbd880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:50.975 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.973+0000 7fbd90341700 1 -- 172.21.15.174:0/3653354361 shutdown_connections 2023-12-25T21:09:50.975 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.973+0000 7fbd90341700 1 -- 172.21.15.174:0/3653354361 wait complete. 2023-12-25T21:09:50.975 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.974+0000 7fbd90341700 1 Processor -- start 2023-12-25T21:09:50.976 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.974+0000 7fbd90341700 1 -- start start 2023-12-25T21:09:50.976 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.974+0000 7fbd90341700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd88060280 0x7fbd880f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:50.976 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.974+0000 7fbd90341700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd880f4ff0 con 0x7fbd88060280 2023-12-25T21:09:50.976 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.974+0000 7fbd8e0dd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd88060280 0x7fbd880f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:50.976 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.974+0000 7fbd8e0dd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd88060280 0x7fbd880f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:37294/0 (socket says 172.21.15.174:37294) 2023-12-25T21:09:50.976 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.974+0000 7fbd8e0dd700 1 -- 172.21.15.174:0/4227903257 learned_addr learned my addr 172.21.15.174:0/4227903257 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:50.977 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.975+0000 7fbd8e0dd700 1 -- 172.21.15.174:0/4227903257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd78005040 con 0x7fbd88060280 2023-12-25T21:09:50.977 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.975+0000 7fbd8e0dd700 1 --2- 172.21.15.174:0/4227903257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd88060280 0x7fbd880f6b70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fbd78013510 tx=0x7fbd7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:50.977 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.975+0000 7fbd7effd700 1 -- 172.21.15.174:0/4227903257 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd78009320 con 0x7fbd88060280 2023-12-25T21:09:50.978 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.975+0000 7fbd90341700 1 -- 172.21.15.174:0/4227903257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd880f5190 con 0x7fbd88060280 2023-12-25T21:09:50.978 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.975+0000 7fbd7effd700 1 -- 172.21.15.174:0/4227903257 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd78013f00 con 0x7fbd88060280 2023-12-25T21:09:50.978 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.975+0000 7fbd7effd700 1 -- 172.21.15.174:0/4227903257 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd78014dc0 con 0x7fbd88060280 2023-12-25T21:09:50.978 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.975+0000 7fbd90341700 1 -- 172.21.15.174:0/4227903257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd880f5620 con 0x7fbd88060280 2023-12-25T21:09:50.978 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.976+0000 7fbd7effd700 1 -- 172.21.15.174:0/4227903257 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd780095f0 con 0x7fbd88060280 2023-12-25T21:09:50.978 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.976+0000 7fbd90341700 1 -- 172.21.15.174:0/4227903257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd88059100 con 0x7fbd88060280 2023-12-25T21:09:50.978 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.976+0000 7fbd7effd700 1 --2- 172.21.15.174:0/4227903257 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbd7402ff60 0x7fbd74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:50.978 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.976+0000 7fbd7effd700 1 -- 172.21.15.174:0/4227903257 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd78041100 con 0x7fbd88060280 2023-12-25T21:09:50.978 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.976+0000 7fbd8d8dc700 1 --2- 172.21.15.174:0/4227903257 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbd7402ff60 0x7fbd74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:50.978 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.977+0000 7fbd8d8dc700 1 --2- 172.21.15.174:0/4227903257 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbd7402ff60 0x7fbd74032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fbd840096a0 tx=0x7fbd84006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:50.981 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:50.981+0000 7fbd7effd700 1 -- 172.21.15.174:0/4227903257 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd78014420 con 0x7fbd88060280 2023-12-25T21:09:51.187 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:51.186+0000 7fbd90341700 1 -- 172.21.15.174:0/4227903257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd880fbf50 con 0x7fbd88060280 2023-12-25T21:09:51.187 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:51.187+0000 7fbd7effd700 1 -- 172.21.15.174:0/4227903257 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd78010230 con 0x7fbd88060280 2023-12-25T21:09:51.188 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:51.188 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:51.189 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:51.188+0000 7fbd90341700 1 -- 172.21.15.174:0/4227903257 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbd7402ff60 msgr2=0x7fbd74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:51.189 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:51.188+0000 7fbd90341700 1 --2- 172.21.15.174:0/4227903257 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbd7402ff60 0x7fbd74032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fbd840096a0 tx=0x7fbd84006b40).stop 2023-12-25T21:09:51.189 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:51.188+0000 7fbd90341700 1 -- 172.21.15.174:0/4227903257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd88060280 msgr2=0x7fbd880f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:51.189 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:51.188+0000 7fbd90341700 1 --2- 172.21.15.174:0/4227903257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd88060280 0x7fbd880f6b70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fbd78013510 tx=0x7fbd7800b040).stop 2023-12-25T21:09:51.189 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:51.188+0000 7fbd90341700 1 -- 172.21.15.174:0/4227903257 shutdown_connections 2023-12-25T21:09:51.189 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:51.188+0000 7fbd90341700 1 --2- 172.21.15.174:0/4227903257 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbd7402ff60 0x7fbd74032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:51.190 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:51.188+0000 7fbd90341700 1 --2- 172.21.15.174:0/4227903257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd88060280 0x7fbd880f6b70 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:51.190 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:51.188+0000 7fbd90341700 1 -- 172.21.15.174:0/4227903257 >> 172.21.15.174:0/4227903257 conn(0x7fbd880f0b70 msgr2=0x7fbd880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:51.190 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:51.188+0000 7fbd90341700 1 -- 172.21.15.174:0/4227903257 shutdown_connections 2023-12-25T21:09:51.190 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:51.188+0000 7fbd90341700 1 -- 172.21.15.174:0/4227903257 wait complete. 2023-12-25T21:09:51.190 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:51.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:51 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:50.594576+0000 mgr.smithi061.jxxeke (mgr.14164) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:51.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:51 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:51.188062+0000 mon.smithi061 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.174:0/4227903257' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:52.571 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:52.571 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:53.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.705+0000 7f8ed0bdb700 1 -- 172.21.15.174:0/2211821443 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ecc0f5410 msgr2=0x7f8ecc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:53.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.705+0000 7f8ed0bdb700 1 --2- 172.21.15.174:0/2211821443 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ecc0f5410 0x7f8ecc0f5830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f8ec0009fb0 tx=0x7f8ec0009b30).stop 2023-12-25T21:09:53.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.705+0000 7f8ed0bdb700 1 -- 172.21.15.174:0/2211821443 shutdown_connections 2023-12-25T21:09:53.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.705+0000 7f8ed0bdb700 1 --2- 172.21.15.174:0/2211821443 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ecc0f5410 0x7f8ecc0f5830 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:53.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.705+0000 7f8ed0bdb700 1 -- 172.21.15.174:0/2211821443 >> 172.21.15.174:0/2211821443 conn(0x7f8ecc0f0b70 msgr2=0x7f8ecc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:53.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.705+0000 7f8ed0bdb700 1 -- 172.21.15.174:0/2211821443 shutdown_connections 2023-12-25T21:09:53.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.705+0000 7f8ed0bdb700 1 -- 172.21.15.174:0/2211821443 wait complete. 2023-12-25T21:09:53.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.706+0000 7f8ed0bdb700 1 Processor -- start 2023-12-25T21:09:53.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.706+0000 7f8ed0bdb700 1 -- start start 2023-12-25T21:09:53.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.706+0000 7f8ed0bdb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ecc0f5410 0x7f8ecc106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:53.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.706+0000 7f8ed0bdb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc106c10 con 0x7f8ecc0f5410 2023-12-25T21:09:53.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.706+0000 7f8eca59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ecc0f5410 0x7f8ecc106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:53.709 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.706+0000 7f8eca59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ecc0f5410 0x7f8ecc106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:37310/0 (socket says 172.21.15.174:37310) 2023-12-25T21:09:53.709 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.706+0000 7f8eca59c700 1 -- 172.21.15.174:0/1212657556 learned_addr learned my addr 172.21.15.174:0/1212657556 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:53.709 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.707+0000 7f8eca59c700 1 -- 172.21.15.174:0/1212657556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec0005040 con 0x7f8ecc0f5410 2023-12-25T21:09:53.709 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.707+0000 7f8eca59c700 1 --2- 172.21.15.174:0/1212657556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ecc0f5410 0x7f8ecc106720 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f8ec0013510 tx=0x7f8ec000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:53.709 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.707+0000 7f8ebaffd700 1 -- 172.21.15.174:0/1212657556 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ec0009320 con 0x7f8ecc0f5410 2023-12-25T21:09:53.709 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.707+0000 7f8ebaffd700 1 -- 172.21.15.174:0/1212657556 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ec0013f00 con 0x7f8ecc0f5410 2023-12-25T21:09:53.709 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.707+0000 7f8ed0bdb700 1 -- 172.21.15.174:0/1212657556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ecc106e10 con 0x7f8ecc0f5410 2023-12-25T21:09:53.709 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.707+0000 7f8ebaffd700 1 -- 172.21.15.174:0/1212657556 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ec0014d90 con 0x7f8ecc0f5410 2023-12-25T21:09:53.709 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.707+0000 7f8ed0bdb700 1 -- 172.21.15.174:0/1212657556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ecc1072a0 con 0x7f8ecc0f5410 2023-12-25T21:09:53.710 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.708+0000 7f8ebaffd700 1 -- 172.21.15.174:0/1212657556 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ec00095f0 con 0x7f8ecc0f5410 2023-12-25T21:09:53.710 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.708+0000 7f8ed0bdb700 1 -- 172.21.15.174:0/1212657556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ecc045ad0 con 0x7f8ecc0f5410 2023-12-25T21:09:53.710 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.708+0000 7f8ebaffd700 1 --2- 172.21.15.174:0/1212657556 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8eb402ff60 0x7f8eb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:53.710 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.708+0000 7f8ebaffd700 1 -- 172.21.15.174:0/1212657556 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ec00410a0 con 0x7f8ecc0f5410 2023-12-25T21:09:53.710 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.708+0000 7f8ec9d9b700 1 --2- 172.21.15.174:0/1212657556 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8eb402ff60 0x7f8eb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:53.710 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.709+0000 7f8ec9d9b700 1 --2- 172.21.15.174:0/1212657556 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8eb402ff60 0x7f8eb4032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f8ebc0096a0 tx=0x7f8ebc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:53.713 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.713+0000 7f8ebaffd700 1 -- 172.21.15.174:0/1212657556 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ec001d7f0 con 0x7f8ecc0f5410 2023-12-25T21:09:53.921 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.920+0000 7f8ed0bdb700 1 -- 172.21.15.174:0/1212657556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ecc108150 con 0x7f8ecc0f5410 2023-12-25T21:09:53.921 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.921+0000 7f8ebaffd700 1 -- 172.21.15.174:0/1212657556 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8ec0010230 con 0x7f8ecc0f5410 2023-12-25T21:09:53.922 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:53.922 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:53.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.922+0000 7f8ed0bdb700 1 -- 172.21.15.174:0/1212657556 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8eb402ff60 msgr2=0x7f8eb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:53.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.922+0000 7f8ed0bdb700 1 --2- 172.21.15.174:0/1212657556 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8eb402ff60 0x7f8eb4032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f8ebc0096a0 tx=0x7f8ebc006b40).stop 2023-12-25T21:09:53.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.922+0000 7f8ed0bdb700 1 -- 172.21.15.174:0/1212657556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ecc0f5410 msgr2=0x7f8ecc106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:53.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.922+0000 7f8ed0bdb700 1 --2- 172.21.15.174:0/1212657556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ecc0f5410 0x7f8ecc106720 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f8ec0013510 tx=0x7f8ec000b040).stop 2023-12-25T21:09:53.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.922+0000 7f8ed0bdb700 1 -- 172.21.15.174:0/1212657556 shutdown_connections 2023-12-25T21:09:53.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.922+0000 7f8ed0bdb700 1 --2- 172.21.15.174:0/1212657556 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8eb402ff60 0x7f8eb4032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:53.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.922+0000 7f8ed0bdb700 1 --2- 172.21.15.174:0/1212657556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ecc0f5410 0x7f8ecc106720 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:53.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.922+0000 7f8ed0bdb700 1 -- 172.21.15.174:0/1212657556 >> 172.21.15.174:0/1212657556 conn(0x7f8ecc0f0b70 msgr2=0x7f8ecc05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:53.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.922+0000 7f8ed0bdb700 1 -- 172.21.15.174:0/1212657556 shutdown_connections 2023-12-25T21:09:53.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:53.923+0000 7f8ed0bdb700 1 -- 172.21.15.174:0/1212657556 wait complete. 2023-12-25T21:09:53.925 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:53.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:52.594908+0000 mgr.smithi061.jxxeke (mgr.14164) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:53.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:52.703800+0000 mon.smithi061 (mon.0) 176 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/mirror_snapshot_schedule"}]: dispatch 2023-12-25T21:09:53.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:52.705450+0000 mon.smithi061 (mon.0) 177 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/trash_purge_schedule"}]: dispatch 2023-12-25T21:09:54.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:54 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:53.922206+0000 mon.smithi061 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.174:0/1212657556' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:55.299 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:55.300 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:55.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:55 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:54.595185+0000 mgr.smithi061.jxxeke (mgr.14164) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:56.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.469+0000 7f4a86b5b700 1 -- 172.21.15.174:0/2791945378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800f5410 msgr2=0x7f4a800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:56.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.469+0000 7f4a86b5b700 1 --2- 172.21.15.174:0/2791945378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800f5410 0x7f4a800f5830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f4a74009fb0 tx=0x7f4a74009b30).stop 2023-12-25T21:09:56.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.469+0000 7f4a86b5b700 1 -- 172.21.15.174:0/2791945378 shutdown_connections 2023-12-25T21:09:56.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.469+0000 7f4a86b5b700 1 --2- 172.21.15.174:0/2791945378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800f5410 0x7f4a800f5830 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:56.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.469+0000 7f4a86b5b700 1 -- 172.21.15.174:0/2791945378 >> 172.21.15.174:0/2791945378 conn(0x7f4a800f0b70 msgr2=0x7f4a800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:56.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.469+0000 7f4a86b5b700 1 -- 172.21.15.174:0/2791945378 shutdown_connections 2023-12-25T21:09:56.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.469+0000 7f4a86b5b700 1 -- 172.21.15.174:0/2791945378 wait complete. 2023-12-25T21:09:56.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.470+0000 7f4a86b5b700 1 Processor -- start 2023-12-25T21:09:56.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.470+0000 7f4a86b5b700 1 -- start start 2023-12-25T21:09:56.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.471+0000 7f4a86b5b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800f5410 0x7f4a80105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:56.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.471+0000 7f4a86b5b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a80105b20 con 0x7f4a800f5410 2023-12-25T21:09:56.473 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.471+0000 7f4a848f7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800f5410 0x7f4a80105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:56.473 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.471+0000 7f4a848f7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800f5410 0x7f4a80105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:38428/0 (socket says 172.21.15.174:38428) 2023-12-25T21:09:56.473 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.471+0000 7f4a848f7700 1 -- 172.21.15.174:0/3447541435 learned_addr learned my addr 172.21.15.174:0/3447541435 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:56.473 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.471+0000 7f4a848f7700 1 -- 172.21.15.174:0/3447541435 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a74005040 con 0x7f4a800f5410 2023-12-25T21:09:56.473 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.471+0000 7f4a848f7700 1 --2- 172.21.15.174:0/3447541435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800f5410 0x7f4a80105630 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f4a74013510 tx=0x7f4a7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:56.473 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.471+0000 7f4a7d7fa700 1 -- 172.21.15.174:0/3447541435 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a74009320 con 0x7f4a800f5410 2023-12-25T21:09:56.473 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.472+0000 7f4a86b5b700 1 -- 172.21.15.174:0/3447541435 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a80105d20 con 0x7f4a800f5410 2023-12-25T21:09:56.473 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.472+0000 7f4a7d7fa700 1 -- 172.21.15.174:0/3447541435 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a74013f00 con 0x7f4a800f5410 2023-12-25T21:09:56.473 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.472+0000 7f4a7d7fa700 1 -- 172.21.15.174:0/3447541435 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a74014d90 con 0x7f4a800f5410 2023-12-25T21:09:56.473 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.472+0000 7f4a86b5b700 1 -- 172.21.15.174:0/3447541435 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a801061b0 con 0x7f4a800f5410 2023-12-25T21:09:56.474 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.472+0000 7f4a7d7fa700 1 -- 172.21.15.174:0/3447541435 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a74014420 con 0x7f4a800f5410 2023-12-25T21:09:56.474 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.472+0000 7f4a86b5b700 1 -- 172.21.15.174:0/3447541435 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a80045ad0 con 0x7f4a800f5410 2023-12-25T21:09:56.474 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.472+0000 7f4a7d7fa700 1 --2- 172.21.15.174:0/3447541435 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f4a7002ff60 0x7f4a70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:56.474 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.473+0000 7f4a7d7fa700 1 -- 172.21.15.174:0/3447541435 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a740410c0 con 0x7f4a800f5410 2023-12-25T21:09:56.474 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.473+0000 7f4a7ffff700 1 --2- 172.21.15.174:0/3447541435 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f4a7002ff60 0x7f4a70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:56.474 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.473+0000 7f4a7ffff700 1 --2- 172.21.15.174:0/3447541435 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f4a7002ff60 0x7f4a70032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f4a6c0096a0 tx=0x7f4a6c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:56.478 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.477+0000 7f4a7d7fa700 1 -- 172.21.15.174:0/3447541435 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a740095f0 con 0x7f4a800f5410 2023-12-25T21:09:56.684 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.683+0000 7f4a86b5b700 1 -- 172.21.15.174:0/3447541435 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a800fae60 con 0x7f4a800f5410 2023-12-25T21:09:56.684 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.684+0000 7f4a7d7fa700 1 -- 172.21.15.174:0/3447541435 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4a7401db70 con 0x7f4a800f5410 2023-12-25T21:09:56.684 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:56.684 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:56.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.685+0000 7f4a86b5b700 1 -- 172.21.15.174:0/3447541435 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f4a7002ff60 msgr2=0x7f4a70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:56.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.685+0000 7f4a86b5b700 1 --2- 172.21.15.174:0/3447541435 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f4a7002ff60 0x7f4a70032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f4a6c0096a0 tx=0x7f4a6c006b40).stop 2023-12-25T21:09:56.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.685+0000 7f4a86b5b700 1 -- 172.21.15.174:0/3447541435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800f5410 msgr2=0x7f4a80105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:56.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.685+0000 7f4a86b5b700 1 --2- 172.21.15.174:0/3447541435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800f5410 0x7f4a80105630 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f4a74013510 tx=0x7f4a7400b040).stop 2023-12-25T21:09:56.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.685+0000 7f4a86b5b700 1 -- 172.21.15.174:0/3447541435 shutdown_connections 2023-12-25T21:09:56.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.685+0000 7f4a86b5b700 1 --2- 172.21.15.174:0/3447541435 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f4a7002ff60 0x7f4a70032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:56.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.685+0000 7f4a86b5b700 1 --2- 172.21.15.174:0/3447541435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800f5410 0x7f4a80105630 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:56.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.685+0000 7f4a86b5b700 1 -- 172.21.15.174:0/3447541435 >> 172.21.15.174:0/3447541435 conn(0x7f4a800f0b70 msgr2=0x7f4a800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:56.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.685+0000 7f4a86b5b700 1 -- 172.21.15.174:0/3447541435 shutdown_connections 2023-12-25T21:09:56.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:56.685+0000 7f4a86b5b700 1 -- 172.21.15.174:0/3447541435 wait complete. 2023-12-25T21:09:56.687 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:09:57.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:57 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:56.685017+0000 mon.smithi061 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.174:0/3447541435' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:09:58.070 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:09:58.070 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:09:58.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:09:58 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:56.595556+0000 mgr.smithi061.jxxeke (mgr.14164) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:09:59.200 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.196+0000 7f9d4a87c700 1 -- 172.21.15.174:0/3317508445 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d44060200 msgr2=0x7f9d440fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:59.200 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.196+0000 7f9d4a87c700 1 --2- 172.21.15.174:0/3317508445 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d44060200 0x7f9d440fb570 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f9d34009fb0 tx=0x7f9d34009b30).stop 2023-12-25T21:09:59.200 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.196+0000 7f9d4a87c700 1 -- 172.21.15.174:0/3317508445 shutdown_connections 2023-12-25T21:09:59.200 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.196+0000 7f9d4a87c700 1 --2- 172.21.15.174:0/3317508445 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d44060200 0x7f9d440fb570 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:59.200 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.196+0000 7f9d4a87c700 1 -- 172.21.15.174:0/3317508445 >> 172.21.15.174:0/3317508445 conn(0x7f9d440f0b70 msgr2=0x7f9d440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:59.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.196+0000 7f9d4a87c700 1 -- 172.21.15.174:0/3317508445 shutdown_connections 2023-12-25T21:09:59.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.197+0000 7f9d4a87c700 1 -- 172.21.15.174:0/3317508445 wait complete. 2023-12-25T21:09:59.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.197+0000 7f9d4a87c700 1 Processor -- start 2023-12-25T21:09:59.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.197+0000 7f9d4a87c700 1 -- start start 2023-12-25T21:09:59.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.197+0000 7f9d4a87c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d44060200 0x7f9d441088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:59.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.197+0000 7f9d4a87c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d44108dc0 con 0x7f9d44060200 2023-12-25T21:09:59.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.197+0000 7f9d43fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d44060200 0x7f9d441088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:59.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.197+0000 7f9d43fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d44060200 0x7f9d441088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:38458/0 (socket says 172.21.15.174:38458) 2023-12-25T21:09:59.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.197+0000 7f9d43fff700 1 -- 172.21.15.174:0/910803858 learned_addr learned my addr 172.21.15.174:0/910803858 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:09:59.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.198+0000 7f9d43fff700 1 -- 172.21.15.174:0/910803858 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d34005040 con 0x7f9d44060200 2023-12-25T21:09:59.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.198+0000 7f9d43fff700 1 --2- 172.21.15.174:0/910803858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d44060200 0x7f9d441088d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f9d34013510 tx=0x7f9d3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:59.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.198+0000 7f9d40ff9700 1 -- 172.21.15.174:0/910803858 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d34009320 con 0x7f9d44060200 2023-12-25T21:09:59.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.198+0000 7f9d40ff9700 1 -- 172.21.15.174:0/910803858 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d34013f00 con 0x7f9d44060200 2023-12-25T21:09:59.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.198+0000 7f9d4a87c700 1 -- 172.21.15.174:0/910803858 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d44108fc0 con 0x7f9d44060200 2023-12-25T21:09:59.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.198+0000 7f9d40ff9700 1 -- 172.21.15.174:0/910803858 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d34014dc0 con 0x7f9d44060200 2023-12-25T21:09:59.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.198+0000 7f9d4a87c700 1 -- 172.21.15.174:0/910803858 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d44109450 con 0x7f9d44060200 2023-12-25T21:09:59.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.199+0000 7f9d40ff9700 1 -- 172.21.15.174:0/910803858 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d340095f0 con 0x7f9d44060200 2023-12-25T21:09:59.203 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.199+0000 7f9d4a87c700 1 -- 172.21.15.174:0/910803858 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d44045ad0 con 0x7f9d44060200 2023-12-25T21:09:59.203 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.199+0000 7f9d40ff9700 1 --2- 172.21.15.174:0/910803858 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f9d2c02ff60 0x7f9d2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:09:59.203 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.199+0000 7f9d40ff9700 1 -- 172.21.15.174:0/910803858 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d340410c0 con 0x7f9d44060200 2023-12-25T21:09:59.203 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.199+0000 7f9d437fe700 1 --2- 172.21.15.174:0/910803858 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f9d2c02ff60 0x7f9d2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:09:59.203 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.200+0000 7f9d437fe700 1 --2- 172.21.15.174:0/910803858 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f9d2c02ff60 0x7f9d2c032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f9d380096a0 tx=0x7f9d38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:09:59.204 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.204+0000 7f9d40ff9700 1 -- 172.21.15.174:0/910803858 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d3401c790 con 0x7f9d44060200 2023-12-25T21:09:59.412 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.411+0000 7f9d4a87c700 1 -- 172.21.15.174:0/910803858 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d4410a300 con 0x7f9d44060200 2023-12-25T21:09:59.412 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.412+0000 7f9d40ff9700 1 -- 172.21.15.174:0/910803858 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9d34010230 con 0x7f9d44060200 2023-12-25T21:09:59.413 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:09:59.413 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:09:59.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.413+0000 7f9d4a87c700 1 -- 172.21.15.174:0/910803858 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f9d2c02ff60 msgr2=0x7f9d2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:59.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.413+0000 7f9d4a87c700 1 --2- 172.21.15.174:0/910803858 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f9d2c02ff60 0x7f9d2c032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f9d380096a0 tx=0x7f9d38006b40).stop 2023-12-25T21:09:59.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.413+0000 7f9d4a87c700 1 -- 172.21.15.174:0/910803858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d44060200 msgr2=0x7f9d441088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:09:59.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.413+0000 7f9d4a87c700 1 --2- 172.21.15.174:0/910803858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d44060200 0x7f9d441088d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f9d34013510 tx=0x7f9d3400b040).stop 2023-12-25T21:09:59.415 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.413+0000 7f9d4a87c700 1 -- 172.21.15.174:0/910803858 shutdown_connections 2023-12-25T21:09:59.415 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.413+0000 7f9d4a87c700 1 --2- 172.21.15.174:0/910803858 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f9d2c02ff60 0x7f9d2c032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:59.415 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.413+0000 7f9d4a87c700 1 --2- 172.21.15.174:0/910803858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d44060200 0x7f9d441088d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:09:59.415 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.413+0000 7f9d4a87c700 1 -- 172.21.15.174:0/910803858 >> 172.21.15.174:0/910803858 conn(0x7f9d440f0b70 msgr2=0x7f9d440f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:09:59.415 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.414+0000 7f9d4a87c700 1 -- 172.21.15.174:0/910803858 shutdown_connections 2023-12-25T21:09:59.415 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:09:59.414+0000 7f9d4a87c700 1 -- 172.21.15.174:0/910803858 wait complete. 2023-12-25T21:09:59.415 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:00.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:00 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:09:58.595831+0000 mgr.smithi061.jxxeke (mgr.14164) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:00.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:00 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:09:59.413291+0000 mon.smithi061 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.174:0/910803858' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:00.814 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:00.815 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:01.946 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.943+0000 7f5d8a673700 1 -- 172.21.15.174:0/743882394 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d840601c0 msgr2=0x7f5d840f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:01.946 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.943+0000 7f5d8a673700 1 --2- 172.21.15.174:0/743882394 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d840601c0 0x7f5d840f9190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f5d74009fb0 tx=0x7f5d74009b30).stop 2023-12-25T21:10:01.946 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.943+0000 7f5d8a673700 1 -- 172.21.15.174:0/743882394 shutdown_connections 2023-12-25T21:10:01.946 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.943+0000 7f5d8a673700 1 --2- 172.21.15.174:0/743882394 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d840601c0 0x7f5d840f9190 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:01.946 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.943+0000 7f5d8a673700 1 -- 172.21.15.174:0/743882394 >> 172.21.15.174:0/743882394 conn(0x7f5d840f0b70 msgr2=0x7f5d840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:01.946 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.943+0000 7f5d8a673700 1 -- 172.21.15.174:0/743882394 shutdown_connections 2023-12-25T21:10:01.947 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.943+0000 7f5d8a673700 1 -- 172.21.15.174:0/743882394 wait complete. 2023-12-25T21:10:01.947 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.944+0000 7f5d8a673700 1 Processor -- start 2023-12-25T21:10:01.947 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.944+0000 7f5d8a673700 1 -- start start 2023-12-25T21:10:01.947 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.944+0000 7f5d8a673700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d840601c0 0x7f5d84106680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:01.947 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.944+0000 7f5d8a673700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d84106b70 con 0x7f5d840601c0 2023-12-25T21:10:01.947 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.944+0000 7f5d83fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d840601c0 0x7f5d84106680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:01.947 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.944+0000 7f5d83fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d840601c0 0x7f5d84106680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:38464/0 (socket says 172.21.15.174:38464) 2023-12-25T21:10:01.947 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.944+0000 7f5d83fff700 1 -- 172.21.15.174:0/2329926812 learned_addr learned my addr 172.21.15.174:0/2329926812 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:01.947 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.945+0000 7f5d83fff700 1 -- 172.21.15.174:0/2329926812 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d74005040 con 0x7f5d840601c0 2023-12-25T21:10:01.947 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.945+0000 7f5d83fff700 1 --2- 172.21.15.174:0/2329926812 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d840601c0 0x7f5d84106680 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f5d74013510 tx=0x7f5d7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:01.947 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.945+0000 7f5d80ff9700 1 -- 172.21.15.174:0/2329926812 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d74009320 con 0x7f5d840601c0 2023-12-25T21:10:01.947 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.945+0000 7f5d80ff9700 1 -- 172.21.15.174:0/2329926812 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d74013f00 con 0x7f5d840601c0 2023-12-25T21:10:01.947 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.945+0000 7f5d8a673700 1 -- 172.21.15.174:0/2329926812 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d84106d70 con 0x7f5d840601c0 2023-12-25T21:10:01.948 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.945+0000 7f5d80ff9700 1 -- 172.21.15.174:0/2329926812 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d74014dc0 con 0x7f5d840601c0 2023-12-25T21:10:01.948 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.945+0000 7f5d8a673700 1 -- 172.21.15.174:0/2329926812 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d84107200 con 0x7f5d840601c0 2023-12-25T21:10:01.948 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.946+0000 7f5d80ff9700 1 -- 172.21.15.174:0/2329926812 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d74014420 con 0x7f5d840601c0 2023-12-25T21:10:01.949 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.946+0000 7f5d80ff9700 1 --2- 172.21.15.174:0/2329926812 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5d6c02ff60 0x7f5d6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:01.949 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.946+0000 7f5d80ff9700 1 -- 172.21.15.174:0/2329926812 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d74041110 con 0x7f5d840601c0 2023-12-25T21:10:01.949 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.946+0000 7f5d837fe700 1 --2- 172.21.15.174:0/2329926812 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5d6c02ff60 0x7f5d6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:01.949 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.946+0000 7f5d8a673700 1 -- 172.21.15.174:0/2329926812 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d84059100 con 0x7f5d840601c0 2023-12-25T21:10:01.949 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.947+0000 7f5d837fe700 1 --2- 172.21.15.174:0/2329926812 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5d6c02ff60 0x7f5d6c032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f5d780096a0 tx=0x7f5d78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:01.952 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:01.952+0000 7f5d80ff9700 1 -- 172.21.15.174:0/2329926812 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d740095f0 con 0x7f5d840601c0 2023-12-25T21:10:02.160 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:02.159+0000 7f5d8a673700 1 -- 172.21.15.174:0/2329926812 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d84045ad0 con 0x7f5d840601c0 2023-12-25T21:10:02.161 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:02.161+0000 7f5d80ff9700 1 -- 172.21.15.174:0/2329926812 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5d7402b020 con 0x7f5d840601c0 2023-12-25T21:10:02.161 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:02.161 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:02.163 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:02.162+0000 7f5d8a673700 1 -- 172.21.15.174:0/2329926812 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5d6c02ff60 msgr2=0x7f5d6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:02.163 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:02.162+0000 7f5d8a673700 1 --2- 172.21.15.174:0/2329926812 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5d6c02ff60 0x7f5d6c032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f5d780096a0 tx=0x7f5d78006b40).stop 2023-12-25T21:10:02.163 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:02.162+0000 7f5d8a673700 1 -- 172.21.15.174:0/2329926812 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d840601c0 msgr2=0x7f5d84106680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:02.163 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:02.162+0000 7f5d8a673700 1 --2- 172.21.15.174:0/2329926812 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d840601c0 0x7f5d84106680 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f5d74013510 tx=0x7f5d7400b040).stop 2023-12-25T21:10:02.163 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:02.162+0000 7f5d8a673700 1 -- 172.21.15.174:0/2329926812 shutdown_connections 2023-12-25T21:10:02.163 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:02.162+0000 7f5d8a673700 1 --2- 172.21.15.174:0/2329926812 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5d6c02ff60 0x7f5d6c032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:02.163 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:02.162+0000 7f5d8a673700 1 --2- 172.21.15.174:0/2329926812 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d840601c0 0x7f5d84106680 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:02.163 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:02.162+0000 7f5d8a673700 1 -- 172.21.15.174:0/2329926812 >> 172.21.15.174:0/2329926812 conn(0x7f5d840f0b70 msgr2=0x7f5d840f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:02.163 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:02.162+0000 7f5d8a673700 1 -- 172.21.15.174:0/2329926812 shutdown_connections 2023-12-25T21:10:02.164 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:02.162+0000 7f5d8a673700 1 -- 172.21.15.174:0/2329926812 wait complete. 2023-12-25T21:10:02.164 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:02.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:02 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:00.596093+0000 mgr.smithi061.jxxeke (mgr.14164) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:02.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:02 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:02.161423+0000 mon.smithi061 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.174:0/2329926812' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:03.545 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:03.545 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:04.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:04 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:02.596320+0000 mgr.smithi061.jxxeke (mgr.14164) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:04.765 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.764+0000 7fc6b4b75700 1 -- 172.21.15.174:0/4097822263 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6b00f6510 msgr2=0x7fc6b00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:04.766 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.764+0000 7fc6b4b75700 1 --2- 172.21.15.174:0/4097822263 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6b00f6510 0x7fc6b00f6930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fc698009fb0 tx=0x7fc698009b30).stop 2023-12-25T21:10:04.766 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.764+0000 7fc6b4b75700 1 -- 172.21.15.174:0/4097822263 shutdown_connections 2023-12-25T21:10:04.766 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.764+0000 7fc6b4b75700 1 --2- 172.21.15.174:0/4097822263 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6b00f6510 0x7fc6b00f6930 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:04.766 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.764+0000 7fc6b4b75700 1 -- 172.21.15.174:0/4097822263 >> 172.21.15.174:0/4097822263 conn(0x7fc6b00f1c70 msgr2=0x7fc6b00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:04.766 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.764+0000 7fc6b4b75700 1 -- 172.21.15.174:0/4097822263 shutdown_connections 2023-12-25T21:10:04.766 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.764+0000 7fc6b4b75700 1 -- 172.21.15.174:0/4097822263 wait complete. 2023-12-25T21:10:04.766 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.764+0000 7fc6b4b75700 1 Processor -- start 2023-12-25T21:10:04.766 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.764+0000 7fc6b4b75700 1 -- start start 2023-12-25T21:10:04.766 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.765+0000 7fc6b4b75700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6b00f6510 0x7fc6b0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:04.766 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.765+0000 7fc6b4b75700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b0106c20 con 0x7fc6b00f6510 2023-12-25T21:10:04.766 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.765+0000 7fc6ae59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6b00f6510 0x7fc6b0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:04.767 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.765+0000 7fc6ae59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6b00f6510 0x7fc6b0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:38490/0 (socket says 172.21.15.174:38490) 2023-12-25T21:10:04.767 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.765+0000 7fc6ae59c700 1 -- 172.21.15.174:0/92036716 learned_addr learned my addr 172.21.15.174:0/92036716 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:04.767 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.765+0000 7fc6ae59c700 1 -- 172.21.15.174:0/92036716 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc698005040 con 0x7fc6b00f6510 2023-12-25T21:10:04.767 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.765+0000 7fc6ae59c700 1 --2- 172.21.15.174:0/92036716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6b00f6510 0x7fc6b0106730 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fc698013510 tx=0x7fc69800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:04.767 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.765+0000 7fc6a6ffd700 1 -- 172.21.15.174:0/92036716 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc698009320 con 0x7fc6b00f6510 2023-12-25T21:10:04.767 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.766+0000 7fc6b4b75700 1 -- 172.21.15.174:0/92036716 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b0106e20 con 0x7fc6b00f6510 2023-12-25T21:10:04.767 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.766+0000 7fc6a6ffd700 1 -- 172.21.15.174:0/92036716 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc698013f00 con 0x7fc6b00f6510 2023-12-25T21:10:04.767 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.766+0000 7fc6a6ffd700 1 -- 172.21.15.174:0/92036716 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc698014d90 con 0x7fc6b00f6510 2023-12-25T21:10:04.767 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.766+0000 7fc6b4b75700 1 -- 172.21.15.174:0/92036716 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b01072b0 con 0x7fc6b00f6510 2023-12-25T21:10:04.768 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.766+0000 7fc6a6ffd700 1 -- 172.21.15.174:0/92036716 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc6980095f0 con 0x7fc6b00f6510 2023-12-25T21:10:04.768 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.766+0000 7fc6a6ffd700 1 --2- 172.21.15.174:0/92036716 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fc69c02ff60 0x7fc69c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:04.768 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.766+0000 7fc6a6ffd700 1 -- 172.21.15.174:0/92036716 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc698041000 con 0x7fc6b00f6510 2023-12-25T21:10:04.768 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.767+0000 7fc6b4b75700 1 -- 172.21.15.174:0/92036716 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b0059100 con 0x7fc6b00f6510 2023-12-25T21:10:04.768 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.767+0000 7fc6add9b700 1 --2- 172.21.15.174:0/92036716 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fc69c02ff60 0x7fc69c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:04.769 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.767+0000 7fc6add9b700 1 --2- 172.21.15.174:0/92036716 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fc69c02ff60 0x7fc69c032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fc6a00096a0 tx=0x7fc6a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:04.771 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.772+0000 7fc6a6ffd700 1 -- 172.21.15.174:0/92036716 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc698014420 con 0x7fc6b00f6510 2023-12-25T21:10:04.979 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.979+0000 7fc6b4b75700 1 -- 172.21.15.174:0/92036716 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6b00fbf60 con 0x7fc6b00f6510 2023-12-25T21:10:04.979 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.980+0000 7fc6a6ffd700 1 -- 172.21.15.174:0/92036716 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc69802b020 con 0x7fc6b00f6510 2023-12-25T21:10:04.980 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:04.980 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:04.981 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.981+0000 7fc6b4b75700 1 -- 172.21.15.174:0/92036716 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fc69c02ff60 msgr2=0x7fc69c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:04.981 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.981+0000 7fc6b4b75700 1 --2- 172.21.15.174:0/92036716 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fc69c02ff60 0x7fc69c032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fc6a00096a0 tx=0x7fc6a0006b40).stop 2023-12-25T21:10:04.981 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.981+0000 7fc6b4b75700 1 -- 172.21.15.174:0/92036716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6b00f6510 msgr2=0x7fc6b0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:04.981 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.981+0000 7fc6b4b75700 1 --2- 172.21.15.174:0/92036716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6b00f6510 0x7fc6b0106730 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fc698013510 tx=0x7fc69800b040).stop 2023-12-25T21:10:04.981 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.981+0000 7fc6b4b75700 1 -- 172.21.15.174:0/92036716 shutdown_connections 2023-12-25T21:10:04.981 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.981+0000 7fc6b4b75700 1 --2- 172.21.15.174:0/92036716 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fc69c02ff60 0x7fc69c032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:04.981 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.981+0000 7fc6b4b75700 1 --2- 172.21.15.174:0/92036716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6b00f6510 0x7fc6b0106730 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:04.982 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.981+0000 7fc6b4b75700 1 -- 172.21.15.174:0/92036716 >> 172.21.15.174:0/92036716 conn(0x7fc6b00f1c70 msgr2=0x7fc6b00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:04.982 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.981+0000 7fc6b4b75700 1 -- 172.21.15.174:0/92036716 shutdown_connections 2023-12-25T21:10:04.982 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:04.981+0000 7fc6b4b75700 1 -- 172.21.15.174:0/92036716 wait complete. 2023-12-25T21:10:04.982 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:05.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:05 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:04.980181+0000 mon.smithi061 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.174:0/92036716' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:06.380 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:06.380 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:06.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:06 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:04.596513+0000 mgr.smithi061.jxxeke (mgr.14164) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:07.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.538+0000 7fd112d64700 1 -- 172.21.15.174:0/662637130 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd10c0f5410 msgr2=0x7fd10c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:07.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.538+0000 7fd112d64700 1 --2- 172.21.15.174:0/662637130 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd10c0f5410 0x7fd10c0f5830 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fd100009fb0 tx=0x7fd100009b30).stop 2023-12-25T21:10:07.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.538+0000 7fd112d64700 1 -- 172.21.15.174:0/662637130 shutdown_connections 2023-12-25T21:10:07.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.538+0000 7fd112d64700 1 --2- 172.21.15.174:0/662637130 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd10c0f5410 0x7fd10c0f5830 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:07.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.538+0000 7fd112d64700 1 -- 172.21.15.174:0/662637130 >> 172.21.15.174:0/662637130 conn(0x7fd10c0f0b70 msgr2=0x7fd10c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:07.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.538+0000 7fd112d64700 1 -- 172.21.15.174:0/662637130 shutdown_connections 2023-12-25T21:10:07.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.538+0000 7fd112d64700 1 -- 172.21.15.174:0/662637130 wait complete. 2023-12-25T21:10:07.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.539+0000 7fd112d64700 1 Processor -- start 2023-12-25T21:10:07.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.539+0000 7fd112d64700 1 -- start start 2023-12-25T21:10:07.541 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.539+0000 7fd112d64700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd10c0f5410 0x7fd10c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:07.541 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.539+0000 7fd112d64700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c060220 con 0x7fd10c0f5410 2023-12-25T21:10:07.541 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.539+0000 7fd110b00700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd10c0f5410 0x7fd10c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:07.541 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.539+0000 7fd110b00700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd10c0f5410 0x7fd10c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:36316/0 (socket says 172.21.15.174:36316) 2023-12-25T21:10:07.541 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.539+0000 7fd110b00700 1 -- 172.21.15.174:0/1145827713 learned_addr learned my addr 172.21.15.174:0/1145827713 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:07.541 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.540+0000 7fd110b00700 1 -- 172.21.15.174:0/1145827713 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd100005040 con 0x7fd10c0f5410 2023-12-25T21:10:07.541 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.540+0000 7fd110b00700 1 --2- 172.21.15.174:0/1145827713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd10c0f5410 0x7fd10c109450 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fd100013510 tx=0x7fd10000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:07.541 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.540+0000 7fd1097fa700 1 -- 172.21.15.174:0/1145827713 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd100009320 con 0x7fd10c0f5410 2023-12-25T21:10:07.541 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.540+0000 7fd112d64700 1 -- 172.21.15.174:0/1145827713 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd10c060420 con 0x7fd10c0f5410 2023-12-25T21:10:07.541 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.540+0000 7fd1097fa700 1 -- 172.21.15.174:0/1145827713 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd100013f00 con 0x7fd10c0f5410 2023-12-25T21:10:07.542 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.540+0000 7fd1097fa700 1 -- 172.21.15.174:0/1145827713 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd100014d90 con 0x7fd10c0f5410 2023-12-25T21:10:07.542 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.540+0000 7fd112d64700 1 -- 172.21.15.174:0/1145827713 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd10c109b00 con 0x7fd10c0f5410 2023-12-25T21:10:07.542 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.541+0000 7fd1097fa700 1 -- 172.21.15.174:0/1145827713 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd1000095f0 con 0x7fd10c0f5410 2023-12-25T21:10:07.542 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.541+0000 7fd112d64700 1 -- 172.21.15.174:0/1145827713 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd10c059100 con 0x7fd10c0f5410 2023-12-25T21:10:07.542 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.541+0000 7fd1097fa700 1 --2- 172.21.15.174:0/1145827713 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd0fc02ff60 0x7fd0fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:07.542 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.541+0000 7fd1097fa700 1 -- 172.21.15.174:0/1145827713 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd1000410a0 con 0x7fd10c0f5410 2023-12-25T21:10:07.542 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.541+0000 7fd10bfff700 1 --2- 172.21.15.174:0/1145827713 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd0fc02ff60 0x7fd0fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:07.542 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.542+0000 7fd10bfff700 1 --2- 172.21.15.174:0/1145827713 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd0fc02ff60 0x7fd0fc032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fd0f80096a0 tx=0x7fd0f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:07.545 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.546+0000 7fd1097fa700 1 -- 172.21.15.174:0/1145827713 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd10001d7f0 con 0x7fd10c0f5410 2023-12-25T21:10:07.751 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.751+0000 7fd112d64700 1 -- 172.21.15.174:0/1145827713 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd10c045ad0 con 0x7fd10c0f5410 2023-12-25T21:10:07.752 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.752+0000 7fd1097fa700 1 -- 172.21.15.174:0/1145827713 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd100010230 con 0x7fd10c0f5410 2023-12-25T21:10:07.752 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:07.752 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:07.753 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.753+0000 7fd112d64700 1 -- 172.21.15.174:0/1145827713 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd0fc02ff60 msgr2=0x7fd0fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:07.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.753+0000 7fd112d64700 1 --2- 172.21.15.174:0/1145827713 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd0fc02ff60 0x7fd0fc032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fd0f80096a0 tx=0x7fd0f8006b40).stop 2023-12-25T21:10:07.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.753+0000 7fd112d64700 1 -- 172.21.15.174:0/1145827713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd10c0f5410 msgr2=0x7fd10c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:07.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.753+0000 7fd112d64700 1 --2- 172.21.15.174:0/1145827713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd10c0f5410 0x7fd10c109450 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fd100013510 tx=0x7fd10000b040).stop 2023-12-25T21:10:07.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.754+0000 7fd112d64700 1 -- 172.21.15.174:0/1145827713 shutdown_connections 2023-12-25T21:10:07.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.754+0000 7fd112d64700 1 --2- 172.21.15.174:0/1145827713 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd0fc02ff60 0x7fd0fc032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:07.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.754+0000 7fd112d64700 1 --2- 172.21.15.174:0/1145827713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd10c0f5410 0x7fd10c109450 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:07.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.754+0000 7fd112d64700 1 -- 172.21.15.174:0/1145827713 >> 172.21.15.174:0/1145827713 conn(0x7fd10c0f0b70 msgr2=0x7fd10c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:07.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.754+0000 7fd112d64700 1 -- 172.21.15.174:0/1145827713 shutdown_connections 2023-12-25T21:10:07.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:07.754+0000 7fd112d64700 1 -- 172.21.15.174:0/1145827713 wait complete. 2023-12-25T21:10:07.754 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:08.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:08 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:06.596807+0000 mgr.smithi061.jxxeke (mgr.14164) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:08.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:08 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:07.752325+0000 mon.smithi061 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.174:0/1145827713' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:09.144 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:09.144 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:10.307 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.305+0000 7f97e0cc4700 1 -- 172.21.15.174:0/4039152234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97dc0f5410 msgr2=0x7f97dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:10.307 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.305+0000 7f97e0cc4700 1 --2- 172.21.15.174:0/4039152234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97dc0f5410 0x7f97dc0f5830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f97c4009fb0 tx=0x7f97c4009b30).stop 2023-12-25T21:10:10.307 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.305+0000 7f97e0cc4700 1 -- 172.21.15.174:0/4039152234 shutdown_connections 2023-12-25T21:10:10.307 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.305+0000 7f97e0cc4700 1 --2- 172.21.15.174:0/4039152234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97dc0f5410 0x7f97dc0f5830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:10.307 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.305+0000 7f97e0cc4700 1 -- 172.21.15.174:0/4039152234 >> 172.21.15.174:0/4039152234 conn(0x7f97dc0f0b70 msgr2=0x7f97dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:10.307 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.305+0000 7f97e0cc4700 1 -- 172.21.15.174:0/4039152234 shutdown_connections 2023-12-25T21:10:10.307 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.305+0000 7f97e0cc4700 1 -- 172.21.15.174:0/4039152234 wait complete. 2023-12-25T21:10:10.307 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.306+0000 7f97e0cc4700 1 Processor -- start 2023-12-25T21:10:10.307 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.306+0000 7f97e0cc4700 1 -- start start 2023-12-25T21:10:10.308 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.306+0000 7f97e0cc4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97dc0f5410 0x7f97dc10b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:10.308 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.306+0000 7f97e0cc4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc107240 con 0x7f97dc0f5410 2023-12-25T21:10:10.308 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.306+0000 7f97da59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97dc0f5410 0x7f97dc10b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:10.308 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.306+0000 7f97da59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97dc0f5410 0x7f97dc10b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:36342/0 (socket says 172.21.15.174:36342) 2023-12-25T21:10:10.308 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.306+0000 7f97da59c700 1 -- 172.21.15.174:0/2456585918 learned_addr learned my addr 172.21.15.174:0/2456585918 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:10.308 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.307+0000 7f97da59c700 1 -- 172.21.15.174:0/2456585918 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c4005040 con 0x7f97dc0f5410 2023-12-25T21:10:10.308 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.307+0000 7f97da59c700 1 --2- 172.21.15.174:0/2456585918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97dc0f5410 0x7f97dc10b850 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f97c4014000 tx=0x7f97c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:10.308 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.307+0000 7f97d2ffd700 1 -- 172.21.15.174:0/2456585918 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97c40095f0 con 0x7f97dc0f5410 2023-12-25T21:10:10.308 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.307+0000 7f97e0cc4700 1 -- 172.21.15.174:0/2456585918 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97dc060220 con 0x7f97dc0f5410 2023-12-25T21:10:10.308 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.307+0000 7f97d2ffd700 1 -- 172.21.15.174:0/2456585918 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f97c4007e70 con 0x7f97dc0f5410 2023-12-25T21:10:10.308 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.307+0000 7f97d2ffd700 1 -- 172.21.15.174:0/2456585918 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97c4014ea0 con 0x7f97dc0f5410 2023-12-25T21:10:10.308 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.307+0000 7f97e0cc4700 1 -- 172.21.15.174:0/2456585918 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97dc10bd40 con 0x7f97dc0f5410 2023-12-25T21:10:10.309 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.308+0000 7f97d2ffd700 1 -- 172.21.15.174:0/2456585918 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f97c4025070 con 0x7f97dc0f5410 2023-12-25T21:10:10.309 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.308+0000 7f97e0cc4700 1 -- 172.21.15.174:0/2456585918 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97dc045ad0 con 0x7f97dc0f5410 2023-12-25T21:10:10.309 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.308+0000 7f97d2ffd700 1 --2- 172.21.15.174:0/2456585918 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f97c802ff60 0x7f97c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:10.310 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.308+0000 7f97d2ffd700 1 -- 172.21.15.174:0/2456585918 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f97c4040a00 con 0x7f97dc0f5410 2023-12-25T21:10:10.310 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.308+0000 7f97d9d9b700 1 --2- 172.21.15.174:0/2456585918 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f97c802ff60 0x7f97c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:10.310 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.309+0000 7f97d9d9b700 1 --2- 172.21.15.174:0/2456585918 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f97c802ff60 0x7f97c8032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f97cc0096a0 tx=0x7f97cc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:10.312 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.313+0000 7f97d2ffd700 1 -- 172.21.15.174:0/2456585918 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f97c40079c0 con 0x7f97dc0f5410 2023-12-25T21:10:10.521 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.521+0000 7f97e0cc4700 1 -- 172.21.15.174:0/2456585918 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f97dc0fae60 con 0x7f97dc0f5410 2023-12-25T21:10:10.522 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.522+0000 7f97d2ffd700 1 -- 172.21.15.174:0/2456585918 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f97c400f300 con 0x7f97dc0f5410 2023-12-25T21:10:10.522 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:10.522 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:10.523 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.523+0000 7f97e0cc4700 1 -- 172.21.15.174:0/2456585918 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f97c802ff60 msgr2=0x7f97c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:10.523 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.523+0000 7f97e0cc4700 1 --2- 172.21.15.174:0/2456585918 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f97c802ff60 0x7f97c8032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f97cc0096a0 tx=0x7f97cc006b40).stop 2023-12-25T21:10:10.523 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.523+0000 7f97e0cc4700 1 -- 172.21.15.174:0/2456585918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97dc0f5410 msgr2=0x7f97dc10b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:10.523 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.523+0000 7f97e0cc4700 1 --2- 172.21.15.174:0/2456585918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97dc0f5410 0x7f97dc10b850 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f97c4014000 tx=0x7f97c400b040).stop 2023-12-25T21:10:10.524 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.524+0000 7f97e0cc4700 1 -- 172.21.15.174:0/2456585918 shutdown_connections 2023-12-25T21:10:10.524 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.524+0000 7f97e0cc4700 1 --2- 172.21.15.174:0/2456585918 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f97c802ff60 0x7f97c8032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:10.524 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.524+0000 7f97e0cc4700 1 --2- 172.21.15.174:0/2456585918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97dc0f5410 0x7f97dc10b850 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:10.524 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.524+0000 7f97e0cc4700 1 -- 172.21.15.174:0/2456585918 >> 172.21.15.174:0/2456585918 conn(0x7f97dc0f0b70 msgr2=0x7f97dc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:10.524 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.524+0000 7f97e0cc4700 1 -- 172.21.15.174:0/2456585918 shutdown_connections 2023-12-25T21:10:10.524 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:10.524+0000 7f97e0cc4700 1 -- 172.21.15.174:0/2456585918 wait complete. 2023-12-25T21:10:10.524 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:10.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:10 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:08.597151+0000 mgr.smithi061.jxxeke (mgr.14164) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:11.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:11 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:10.522360+0000 mon.smithi061 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.174:0/2456585918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:11.932 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:11.932 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:12.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:12 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:10.597452+0000 mgr.smithi061.jxxeke (mgr.14164) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:13.067 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.067+0000 7fa8741db700 1 -- 172.21.15.174:0/3131062134 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa86c0f5410 msgr2=0x7fa86c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:13.067 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.067+0000 7fa8741db700 1 --2- 172.21.15.174:0/3131062134 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa86c0f5410 0x7fa86c0f5830 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fa868009fb0 tx=0x7fa868009b30).stop 2023-12-25T21:10:13.067 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.067+0000 7fa8741db700 1 -- 172.21.15.174:0/3131062134 shutdown_connections 2023-12-25T21:10:13.068 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.067+0000 7fa8741db700 1 --2- 172.21.15.174:0/3131062134 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa86c0f5410 0x7fa86c0f5830 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:13.068 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.067+0000 7fa8741db700 1 -- 172.21.15.174:0/3131062134 >> 172.21.15.174:0/3131062134 conn(0x7fa86c0f0b70 msgr2=0x7fa86c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:13.068 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.067+0000 7fa8741db700 1 -- 172.21.15.174:0/3131062134 shutdown_connections 2023-12-25T21:10:13.068 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.067+0000 7fa8741db700 1 -- 172.21.15.174:0/3131062134 wait complete. 2023-12-25T21:10:13.070 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.068+0000 7fa8741db700 1 Processor -- start 2023-12-25T21:10:13.070 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.068+0000 7fa8741db700 1 -- start start 2023-12-25T21:10:13.070 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.068+0000 7fa8741db700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa86c0f5410 0x7fa86c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:13.070 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.068+0000 7fa8741db700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c105b20 con 0x7fa86c0f5410 2023-12-25T21:10:13.070 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.068+0000 7fa871f77700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa86c0f5410 0x7fa86c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:13.070 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.068+0000 7fa871f77700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa86c0f5410 0x7fa86c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:36350/0 (socket says 172.21.15.174:36350) 2023-12-25T21:10:13.071 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.068+0000 7fa871f77700 1 -- 172.21.15.174:0/2281760769 learned_addr learned my addr 172.21.15.174:0/2281760769 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:13.071 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.069+0000 7fa871f77700 1 -- 172.21.15.174:0/2281760769 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa868005040 con 0x7fa86c0f5410 2023-12-25T21:10:13.071 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.069+0000 7fa871f77700 1 --2- 172.21.15.174:0/2281760769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa86c0f5410 0x7fa86c105630 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fa868013510 tx=0x7fa86800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:13.071 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.069+0000 7fa8627fc700 1 -- 172.21.15.174:0/2281760769 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa868009320 con 0x7fa86c0f5410 2023-12-25T21:10:13.071 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.069+0000 7fa8741db700 1 -- 172.21.15.174:0/2281760769 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa86c105d20 con 0x7fa86c0f5410 2023-12-25T21:10:13.071 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.069+0000 7fa8627fc700 1 -- 172.21.15.174:0/2281760769 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa868013f00 con 0x7fa86c0f5410 2023-12-25T21:10:13.071 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.069+0000 7fa8627fc700 1 -- 172.21.15.174:0/2281760769 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa868014d90 con 0x7fa86c0f5410 2023-12-25T21:10:13.071 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.069+0000 7fa8741db700 1 -- 172.21.15.174:0/2281760769 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa86c1061b0 con 0x7fa86c0f5410 2023-12-25T21:10:13.071 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.070+0000 7fa8627fc700 1 -- 172.21.15.174:0/2281760769 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8680095f0 con 0x7fa86c0f5410 2023-12-25T21:10:13.071 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.070+0000 7fa8741db700 1 -- 172.21.15.174:0/2281760769 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86c045ad0 con 0x7fa86c0f5410 2023-12-25T21:10:13.071 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.070+0000 7fa8627fc700 1 --2- 172.21.15.174:0/2281760769 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa85802ffb0 0x7fa858032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:13.071 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.070+0000 7fa8627fc700 1 -- 172.21.15.174:0/2281760769 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8680410c0 con 0x7fa86c0f5410 2023-12-25T21:10:13.072 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.070+0000 7fa871776700 1 --2- 172.21.15.174:0/2281760769 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa85802ffb0 0x7fa858032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:13.072 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.071+0000 7fa871776700 1 --2- 172.21.15.174:0/2281760769 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa85802ffb0 0x7fa858032470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fa85c0096a0 tx=0x7fa85c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:13.074 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.075+0000 7fa8627fc700 1 -- 172.21.15.174:0/2281760769 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa868014420 con 0x7fa86c0f5410 2023-12-25T21:10:13.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.285+0000 7fa8741db700 1 -- 172.21.15.174:0/2281760769 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa86c0fae60 con 0x7fa86c0f5410 2023-12-25T21:10:13.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.285+0000 7fa8627fc700 1 -- 172.21.15.174:0/2281760769 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa86801d7f0 con 0x7fa86c0f5410 2023-12-25T21:10:13.285 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:13.285 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:13.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.287+0000 7fa8741db700 1 -- 172.21.15.174:0/2281760769 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa85802ffb0 msgr2=0x7fa858032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:13.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.287+0000 7fa8741db700 1 --2- 172.21.15.174:0/2281760769 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa85802ffb0 0x7fa858032470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fa85c0096a0 tx=0x7fa85c006b40).stop 2023-12-25T21:10:13.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.287+0000 7fa8741db700 1 -- 172.21.15.174:0/2281760769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa86c0f5410 msgr2=0x7fa86c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:13.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.287+0000 7fa8741db700 1 --2- 172.21.15.174:0/2281760769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa86c0f5410 0x7fa86c105630 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fa868013510 tx=0x7fa86800b040).stop 2023-12-25T21:10:13.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.287+0000 7fa8741db700 1 -- 172.21.15.174:0/2281760769 shutdown_connections 2023-12-25T21:10:13.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.287+0000 7fa8741db700 1 --2- 172.21.15.174:0/2281760769 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa85802ffb0 0x7fa858032470 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:13.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.287+0000 7fa8741db700 1 --2- 172.21.15.174:0/2281760769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa86c0f5410 0x7fa86c105630 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:13.288 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.287+0000 7fa8741db700 1 -- 172.21.15.174:0/2281760769 >> 172.21.15.174:0/2281760769 conn(0x7fa86c0f0b70 msgr2=0x7fa86c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:13.288 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.287+0000 7fa8741db700 1 -- 172.21.15.174:0/2281760769 shutdown_connections 2023-12-25T21:10:13.288 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:13.287+0000 7fa8741db700 1 -- 172.21.15.174:0/2281760769 wait complete. 2023-12-25T21:10:13.288 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:13.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:13 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:13.285761+0000 mon.smithi061 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.174:0/2281760769' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:14.658 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:14.659 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:14.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:14 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:12.597758+0000 mgr.smithi061.jxxeke (mgr.14164) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:15.778 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.775+0000 7ff65f213700 1 -- 172.21.15.174:0/2780952504 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6580f5410 msgr2=0x7ff6580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:15.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.775+0000 7ff65f213700 1 --2- 172.21.15.174:0/2780952504 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6580f5410 0x7ff6580f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7ff64c009fb0 tx=0x7ff64c009b30).stop 2023-12-25T21:10:15.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.775+0000 7ff65f213700 1 -- 172.21.15.174:0/2780952504 shutdown_connections 2023-12-25T21:10:15.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.775+0000 7ff65f213700 1 --2- 172.21.15.174:0/2780952504 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6580f5410 0x7ff6580f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:15.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.775+0000 7ff65f213700 1 -- 172.21.15.174:0/2780952504 >> 172.21.15.174:0/2780952504 conn(0x7ff6580f0b70 msgr2=0x7ff6580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:15.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.775+0000 7ff65f213700 1 -- 172.21.15.174:0/2780952504 shutdown_connections 2023-12-25T21:10:15.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.775+0000 7ff65f213700 1 -- 172.21.15.174:0/2780952504 wait complete. 2023-12-25T21:10:15.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.776+0000 7ff65f213700 1 Processor -- start 2023-12-25T21:10:15.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.776+0000 7ff65f213700 1 -- start start 2023-12-25T21:10:15.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.776+0000 7ff65f213700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6580f5410 0x7ff658106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:15.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.776+0000 7ff65f213700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff658106c20 con 0x7ff6580f5410 2023-12-25T21:10:15.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.776+0000 7ff65cfaf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6580f5410 0x7ff658106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:15.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.776+0000 7ff65cfaf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6580f5410 0x7ff658106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:36360/0 (socket says 172.21.15.174:36360) 2023-12-25T21:10:15.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.776+0000 7ff65cfaf700 1 -- 172.21.15.174:0/1132128605 learned_addr learned my addr 172.21.15.174:0/1132128605 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:15.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.777+0000 7ff65cfaf700 1 -- 172.21.15.174:0/1132128605 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64c005040 con 0x7ff6580f5410 2023-12-25T21:10:15.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.777+0000 7ff65cfaf700 1 --2- 172.21.15.174:0/1132128605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6580f5410 0x7ff658106730 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7ff64c013510 tx=0x7ff64c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:15.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.777+0000 7ff6557fa700 1 -- 172.21.15.174:0/1132128605 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff64c009320 con 0x7ff6580f5410 2023-12-25T21:10:15.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.777+0000 7ff65f213700 1 -- 172.21.15.174:0/1132128605 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff658106e20 con 0x7ff6580f5410 2023-12-25T21:10:15.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.777+0000 7ff6557fa700 1 -- 172.21.15.174:0/1132128605 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff64c013f00 con 0x7ff6580f5410 2023-12-25T21:10:15.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.777+0000 7ff6557fa700 1 -- 172.21.15.174:0/1132128605 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff64c014d90 con 0x7ff6580f5410 2023-12-25T21:10:15.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.777+0000 7ff65f213700 1 -- 172.21.15.174:0/1132128605 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6581072b0 con 0x7ff6580f5410 2023-12-25T21:10:15.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.778+0000 7ff6557fa700 1 -- 172.21.15.174:0/1132128605 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff64c0263a0 con 0x7ff6580f5410 2023-12-25T21:10:15.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.778+0000 7ff6557fa700 1 --2- 172.21.15.174:0/1132128605 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff64802ff60 0x7ff648032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:15.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.778+0000 7ff6557fa700 1 -- 172.21.15.174:0/1132128605 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff64c0410a0 con 0x7ff6580f5410 2023-12-25T21:10:15.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.778+0000 7ff65f213700 1 -- 172.21.15.174:0/1132128605 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff658045ad0 con 0x7ff6580f5410 2023-12-25T21:10:15.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.778+0000 7ff657fff700 1 --2- 172.21.15.174:0/1132128605 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff64802ff60 0x7ff648032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:15.781 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.779+0000 7ff657fff700 1 --2- 172.21.15.174:0/1132128605 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff64802ff60 0x7ff648032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff6440096a0 tx=0x7ff644006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:15.782 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.783+0000 7ff6557fa700 1 -- 172.21.15.174:0/1132128605 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff64c012920 con 0x7ff6580f5410 2023-12-25T21:10:15.988 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.988+0000 7ff65f213700 1 -- 172.21.15.174:0/1132128605 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff658107ff0 con 0x7ff6580f5410 2023-12-25T21:10:15.989 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.989+0000 7ff6557fa700 1 -- 172.21.15.174:0/1132128605 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff64c02b020 con 0x7ff6580f5410 2023-12-25T21:10:15.989 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:15.989 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:15.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.991+0000 7ff65f213700 1 -- 172.21.15.174:0/1132128605 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff64802ff60 msgr2=0x7ff648032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:15.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.991+0000 7ff65f213700 1 --2- 172.21.15.174:0/1132128605 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff64802ff60 0x7ff648032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff6440096a0 tx=0x7ff644006b40).stop 2023-12-25T21:10:15.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.991+0000 7ff65f213700 1 -- 172.21.15.174:0/1132128605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6580f5410 msgr2=0x7ff658106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:15.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.991+0000 7ff65f213700 1 --2- 172.21.15.174:0/1132128605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6580f5410 0x7ff658106730 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7ff64c013510 tx=0x7ff64c00b040).stop 2023-12-25T21:10:15.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.991+0000 7ff65f213700 1 -- 172.21.15.174:0/1132128605 shutdown_connections 2023-12-25T21:10:15.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.991+0000 7ff65f213700 1 --2- 172.21.15.174:0/1132128605 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff64802ff60 0x7ff648032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:15.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.991+0000 7ff65f213700 1 --2- 172.21.15.174:0/1132128605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6580f5410 0x7ff658106730 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:15.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.991+0000 7ff65f213700 1 -- 172.21.15.174:0/1132128605 >> 172.21.15.174:0/1132128605 conn(0x7ff6580f0b70 msgr2=0x7ff6580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:15.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.991+0000 7ff65f213700 1 -- 172.21.15.174:0/1132128605 shutdown_connections 2023-12-25T21:10:15.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:15.991+0000 7ff65f213700 1 -- 172.21.15.174:0/1132128605 wait complete. 2023-12-25T21:10:15.992 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:16.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:16 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:14.598121+0000 mgr.smithi061.jxxeke (mgr.14164) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:16.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:16 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:15.989711+0000 mon.smithi061 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.174:0/1132128605' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:17.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:17.368 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:18.593 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.591+0000 7faad0567700 1 -- 172.21.15.174:0/2875723755 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faac80f5410 msgr2=0x7faac80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:18.594 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.591+0000 7faad0567700 1 --2- 172.21.15.174:0/2875723755 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faac80f5410 0x7faac80f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7faab8009fb0 tx=0x7faab8009b30).stop 2023-12-25T21:10:18.594 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.591+0000 7faad0567700 1 -- 172.21.15.174:0/2875723755 shutdown_connections 2023-12-25T21:10:18.594 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.591+0000 7faad0567700 1 --2- 172.21.15.174:0/2875723755 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faac80f5410 0x7faac80f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:18.594 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.591+0000 7faad0567700 1 -- 172.21.15.174:0/2875723755 >> 172.21.15.174:0/2875723755 conn(0x7faac80f0b70 msgr2=0x7faac80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:18.594 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.591+0000 7faad0567700 1 -- 172.21.15.174:0/2875723755 shutdown_connections 2023-12-25T21:10:18.594 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.591+0000 7faad0567700 1 -- 172.21.15.174:0/2875723755 wait complete. 2023-12-25T21:10:18.594 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.592+0000 7faad0567700 1 Processor -- start 2023-12-25T21:10:18.594 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.592+0000 7faad0567700 1 -- start start 2023-12-25T21:10:18.594 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.592+0000 7faad0567700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faac80f5410 0x7faac8105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:18.594 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.592+0000 7faad0567700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac8105af0 con 0x7faac80f5410 2023-12-25T21:10:18.594 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.592+0000 7faace303700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faac80f5410 0x7faac8105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:18.595 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.592+0000 7faace303700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faac80f5410 0x7faac8105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:56940/0 (socket says 172.21.15.174:56940) 2023-12-25T21:10:18.595 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.592+0000 7faace303700 1 -- 172.21.15.174:0/3045699844 learned_addr learned my addr 172.21.15.174:0/3045699844 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:18.595 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.593+0000 7faace303700 1 -- 172.21.15.174:0/3045699844 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab8005040 con 0x7faac80f5410 2023-12-25T21:10:18.595 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.593+0000 7faace303700 1 --2- 172.21.15.174:0/3045699844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faac80f5410 0x7faac8105600 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7faab8013510 tx=0x7faab800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:18.595 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.593+0000 7faabeffd700 1 -- 172.21.15.174:0/3045699844 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faab8009320 con 0x7faac80f5410 2023-12-25T21:10:18.595 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.593+0000 7faabeffd700 1 -- 172.21.15.174:0/3045699844 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faab8013f00 con 0x7faac80f5410 2023-12-25T21:10:18.595 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.593+0000 7faabeffd700 1 -- 172.21.15.174:0/3045699844 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faab8014d90 con 0x7faac80f5410 2023-12-25T21:10:18.595 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.593+0000 7faad0567700 1 -- 172.21.15.174:0/3045699844 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faac8105cf0 con 0x7faac80f5410 2023-12-25T21:10:18.595 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.593+0000 7faad0567700 1 -- 172.21.15.174:0/3045699844 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faac8106180 con 0x7faac80f5410 2023-12-25T21:10:18.596 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.594+0000 7faabeffd700 1 -- 172.21.15.174:0/3045699844 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faab8014420 con 0x7faac80f5410 2023-12-25T21:10:18.596 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.594+0000 7faad0567700 1 -- 172.21.15.174:0/3045699844 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faac8045ad0 con 0x7faac80f5410 2023-12-25T21:10:18.596 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.594+0000 7faabeffd700 1 --2- 172.21.15.174:0/3045699844 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7faab402ff60 0x7faab4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:18.596 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.594+0000 7faabeffd700 1 -- 172.21.15.174:0/3045699844 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faab8040ff0 con 0x7faac80f5410 2023-12-25T21:10:18.596 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.594+0000 7faacdb02700 1 --2- 172.21.15.174:0/3045699844 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7faab402ff60 0x7faab4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:18.596 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.595+0000 7faacdb02700 1 --2- 172.21.15.174:0/3045699844 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7faab402ff60 0x7faab4032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7faac40096a0 tx=0x7faac4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:18.599 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.599+0000 7faabeffd700 1 -- 172.21.15.174:0/3045699844 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faab80095f0 con 0x7faac80f5410 2023-12-25T21:10:18.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:18 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:16.598469+0000 mgr.smithi061.jxxeke (mgr.14164) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:18.807 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.807+0000 7faad0567700 1 -- 172.21.15.174:0/3045699844 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faac80fae60 con 0x7faac80f5410 2023-12-25T21:10:18.808 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.807+0000 7faabeffd700 1 -- 172.21.15.174:0/3045699844 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faab801db70 con 0x7faac80f5410 2023-12-25T21:10:18.808 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:18.808 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:18.810 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.809+0000 7faad0567700 1 -- 172.21.15.174:0/3045699844 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7faab402ff60 msgr2=0x7faab4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:18.810 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.809+0000 7faad0567700 1 --2- 172.21.15.174:0/3045699844 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7faab402ff60 0x7faab4032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7faac40096a0 tx=0x7faac4006b40).stop 2023-12-25T21:10:18.810 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.809+0000 7faad0567700 1 -- 172.21.15.174:0/3045699844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faac80f5410 msgr2=0x7faac8105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:18.810 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.809+0000 7faad0567700 1 --2- 172.21.15.174:0/3045699844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faac80f5410 0x7faac8105600 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7faab8013510 tx=0x7faab800b040).stop 2023-12-25T21:10:18.810 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.809+0000 7faad0567700 1 -- 172.21.15.174:0/3045699844 shutdown_connections 2023-12-25T21:10:18.810 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.809+0000 7faad0567700 1 --2- 172.21.15.174:0/3045699844 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7faab402ff60 0x7faab4032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:18.810 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.809+0000 7faad0567700 1 --2- 172.21.15.174:0/3045699844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faac80f5410 0x7faac8105600 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:18.810 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.809+0000 7faad0567700 1 -- 172.21.15.174:0/3045699844 >> 172.21.15.174:0/3045699844 conn(0x7faac80f0b70 msgr2=0x7faac80f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:18.810 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.809+0000 7faad0567700 1 -- 172.21.15.174:0/3045699844 shutdown_connections 2023-12-25T21:10:18.811 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:18.809+0000 7faad0567700 1 -- 172.21.15.174:0/3045699844 wait complete. 2023-12-25T21:10:18.811 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:19.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:18.808708+0000 mon.smithi061 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.174:0/3045699844' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:20.218 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:20.219 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:20.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:20 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:18.598739+0000 mgr.smithi061.jxxeke (mgr.14164) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:21.425 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.423+0000 7f1ff9b8e700 1 -- 172.21.15.174:0/2556080560 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ff40f5410 msgr2=0x7f1ff40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:21.425 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.423+0000 7f1ff9b8e700 1 --2- 172.21.15.174:0/2556080560 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ff40f5410 0x7f1ff40f5830 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f1fe4009fb0 tx=0x7f1fe4009b30).stop 2023-12-25T21:10:21.425 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.423+0000 7f1ff9b8e700 1 -- 172.21.15.174:0/2556080560 shutdown_connections 2023-12-25T21:10:21.425 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.423+0000 7f1ff9b8e700 1 --2- 172.21.15.174:0/2556080560 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ff40f5410 0x7f1ff40f5830 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:21.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.423+0000 7f1ff9b8e700 1 -- 172.21.15.174:0/2556080560 >> 172.21.15.174:0/2556080560 conn(0x7f1ff40f0b70 msgr2=0x7f1ff40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:21.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.423+0000 7f1ff9b8e700 1 -- 172.21.15.174:0/2556080560 shutdown_connections 2023-12-25T21:10:21.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.423+0000 7f1ff9b8e700 1 -- 172.21.15.174:0/2556080560 wait complete. 2023-12-25T21:10:21.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.424+0000 7f1ff9b8e700 1 Processor -- start 2023-12-25T21:10:21.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.424+0000 7f1ff9b8e700 1 -- start start 2023-12-25T21:10:21.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.424+0000 7f1ff9b8e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ff40f5410 0x7f1ff4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:21.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.424+0000 7f1ff9b8e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff4105b20 con 0x7f1ff40f5410 2023-12-25T21:10:21.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.424+0000 7f1ff37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ff40f5410 0x7f1ff4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:21.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.424+0000 7f1ff37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ff40f5410 0x7f1ff4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:56966/0 (socket says 172.21.15.174:56966) 2023-12-25T21:10:21.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.424+0000 7f1ff37fe700 1 -- 172.21.15.174:0/1109790464 learned_addr learned my addr 172.21.15.174:0/1109790464 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:21.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.425+0000 7f1ff37fe700 1 -- 172.21.15.174:0/1109790464 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe4005040 con 0x7f1ff40f5410 2023-12-25T21:10:21.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.425+0000 7f1ff37fe700 1 --2- 172.21.15.174:0/1109790464 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ff40f5410 0x7f1ff4105630 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f1fe4013510 tx=0x7f1fe400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:21.428 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.425+0000 7f1ff0ff9700 1 -- 172.21.15.174:0/1109790464 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fe4009320 con 0x7f1ff40f5410 2023-12-25T21:10:21.428 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.425+0000 7f1ff9b8e700 1 -- 172.21.15.174:0/1109790464 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff4105d20 con 0x7f1ff40f5410 2023-12-25T21:10:21.428 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.425+0000 7f1ff0ff9700 1 -- 172.21.15.174:0/1109790464 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1fe4013f00 con 0x7f1ff40f5410 2023-12-25T21:10:21.428 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.425+0000 7f1ff0ff9700 1 -- 172.21.15.174:0/1109790464 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fe4014d90 con 0x7f1ff40f5410 2023-12-25T21:10:21.428 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.425+0000 7f1ff9b8e700 1 -- 172.21.15.174:0/1109790464 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff41061b0 con 0x7f1ff40f5410 2023-12-25T21:10:21.428 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.426+0000 7f1ff0ff9700 1 -- 172.21.15.174:0/1109790464 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1fe4014420 con 0x7f1ff40f5410 2023-12-25T21:10:21.428 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.426+0000 7f1ff9b8e700 1 -- 172.21.15.174:0/1109790464 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff4045ad0 con 0x7f1ff40f5410 2023-12-25T21:10:21.428 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.426+0000 7f1ff0ff9700 1 --2- 172.21.15.174:0/1109790464 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1fe002ff60 0x7f1fe0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:21.428 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.426+0000 7f1ff0ff9700 1 -- 172.21.15.174:0/1109790464 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1fe4041070 con 0x7f1ff40f5410 2023-12-25T21:10:21.428 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.426+0000 7f1feaffd700 1 --2- 172.21.15.174:0/1109790464 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1fe002ff60 0x7f1fe0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:21.428 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.427+0000 7f1feaffd700 1 --2- 172.21.15.174:0/1109790464 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1fe002ff60 0x7f1fe0032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f1fdc0096a0 tx=0x7f1fdc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:21.431 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.431+0000 7f1ff0ff9700 1 -- 172.21.15.174:0/1109790464 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1fe406d2b0 con 0x7f1ff40f5410 2023-12-25T21:10:21.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.633+0000 7f1ff9b8e700 1 -- 172.21.15.174:0/1109790464 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ff40fae60 con 0x7f1ff40f5410 2023-12-25T21:10:21.635 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.634+0000 7f1ff0ff9700 1 -- 172.21.15.174:0/1109790464 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1fe4012330 con 0x7f1ff40f5410 2023-12-25T21:10:21.635 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:21.635 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:21.636 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.635+0000 7f1ff9b8e700 1 -- 172.21.15.174:0/1109790464 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1fe002ff60 msgr2=0x7f1fe0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:21.636 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.635+0000 7f1ff9b8e700 1 --2- 172.21.15.174:0/1109790464 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1fe002ff60 0x7f1fe0032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f1fdc0096a0 tx=0x7f1fdc006b40).stop 2023-12-25T21:10:21.636 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.635+0000 7f1ff9b8e700 1 -- 172.21.15.174:0/1109790464 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ff40f5410 msgr2=0x7f1ff4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:21.637 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.635+0000 7f1ff9b8e700 1 --2- 172.21.15.174:0/1109790464 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ff40f5410 0x7f1ff4105630 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f1fe4013510 tx=0x7f1fe400b040).stop 2023-12-25T21:10:21.637 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.636+0000 7f1ff9b8e700 1 -- 172.21.15.174:0/1109790464 shutdown_connections 2023-12-25T21:10:21.637 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.636+0000 7f1ff9b8e700 1 --2- 172.21.15.174:0/1109790464 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1fe002ff60 0x7f1fe0032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:21.637 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.636+0000 7f1ff9b8e700 1 --2- 172.21.15.174:0/1109790464 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ff40f5410 0x7f1ff4105630 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:21.637 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.636+0000 7f1ff9b8e700 1 -- 172.21.15.174:0/1109790464 >> 172.21.15.174:0/1109790464 conn(0x7f1ff40f0b70 msgr2=0x7f1ff40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:21.637 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.636+0000 7f1ff9b8e700 1 -- 172.21.15.174:0/1109790464 shutdown_connections 2023-12-25T21:10:21.637 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:21.636+0000 7f1ff9b8e700 1 -- 172.21.15.174:0/1109790464 wait complete. 2023-12-25T21:10:21.637 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:22.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:22 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:20.599104+0000 mgr.smithi061.jxxeke (mgr.14164) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:22.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:22 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:21.635407+0000 mon.smithi061 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.174:0/1109790464' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:23.027 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:23.028 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:24.237 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.234+0000 7f6100941700 1 -- 172.21.15.174:0/681455688 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60fc0f5410 msgr2=0x7f60fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:24.237 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.234+0000 7f6100941700 1 --2- 172.21.15.174:0/681455688 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60fc0f5410 0x7f60fc0f5830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f60ec009fb0 tx=0x7f60ec009b30).stop 2023-12-25T21:10:24.237 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.235+0000 7f6100941700 1 -- 172.21.15.174:0/681455688 shutdown_connections 2023-12-25T21:10:24.237 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.235+0000 7f6100941700 1 --2- 172.21.15.174:0/681455688 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60fc0f5410 0x7f60fc0f5830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:24.237 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.235+0000 7f6100941700 1 -- 172.21.15.174:0/681455688 >> 172.21.15.174:0/681455688 conn(0x7f60fc0f0b70 msgr2=0x7f60fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:24.237 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.235+0000 7f6100941700 1 -- 172.21.15.174:0/681455688 shutdown_connections 2023-12-25T21:10:24.238 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.235+0000 7f6100941700 1 -- 172.21.15.174:0/681455688 wait complete. 2023-12-25T21:10:24.238 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.235+0000 7f6100941700 1 Processor -- start 2023-12-25T21:10:24.238 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.235+0000 7f6100941700 1 -- start start 2023-12-25T21:10:24.238 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.235+0000 7f6100941700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60fc0f5410 0x7f60fc0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:24.238 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.235+0000 7f6100941700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc0fb820 con 0x7f60fc0f5410 2023-12-25T21:10:24.238 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.235+0000 7f60fa59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60fc0f5410 0x7f60fc0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:24.238 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.236+0000 7f60fa59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60fc0f5410 0x7f60fc0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:56980/0 (socket says 172.21.15.174:56980) 2023-12-25T21:10:24.238 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.236+0000 7f60fa59c700 1 -- 172.21.15.174:0/2489462063 learned_addr learned my addr 172.21.15.174:0/2489462063 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:24.238 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.236+0000 7f60fa59c700 1 -- 172.21.15.174:0/2489462063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ec005040 con 0x7f60fc0f5410 2023-12-25T21:10:24.238 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.236+0000 7f60fa59c700 1 --2- 172.21.15.174:0/2489462063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60fc0f5410 0x7f60fc0fd3a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f60ec013510 tx=0x7f60ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:24.238 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.236+0000 7f60eaffd700 1 -- 172.21.15.174:0/2489462063 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60ec009320 con 0x7f60fc0f5410 2023-12-25T21:10:24.238 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.236+0000 7f6100941700 1 -- 172.21.15.174:0/2489462063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60fc0fb9c0 con 0x7f60fc0f5410 2023-12-25T21:10:24.238 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.236+0000 7f60eaffd700 1 -- 172.21.15.174:0/2489462063 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f60ec013f00 con 0x7f60fc0f5410 2023-12-25T21:10:24.238 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.236+0000 7f60eaffd700 1 -- 172.21.15.174:0/2489462063 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60ec014d90 con 0x7f60fc0f5410 2023-12-25T21:10:24.239 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.236+0000 7f6100941700 1 -- 172.21.15.174:0/2489462063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60fc0fbe50 con 0x7f60fc0f5410 2023-12-25T21:10:24.239 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.237+0000 7f60eaffd700 1 -- 172.21.15.174:0/2489462063 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60ec014420 con 0x7f60fc0f5410 2023-12-25T21:10:24.240 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.237+0000 7f6100941700 1 -- 172.21.15.174:0/2489462063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60fc059100 con 0x7f60fc0f5410 2023-12-25T21:10:24.240 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.237+0000 7f60eaffd700 1 --2- 172.21.15.174:0/2489462063 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f60e402ff60 0x7f60e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:24.240 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.237+0000 7f60eaffd700 1 -- 172.21.15.174:0/2489462063 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f60ec041050 con 0x7f60fc0f5410 2023-12-25T21:10:24.240 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.237+0000 7f60f9d9b700 1 --2- 172.21.15.174:0/2489462063 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f60e402ff60 0x7f60e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:24.240 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.238+0000 7f60f9d9b700 1 --2- 172.21.15.174:0/2489462063 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f60e402ff60 0x7f60e4032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f60f00096a0 tx=0x7f60f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:24.242 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.242+0000 7f60eaffd700 1 -- 172.21.15.174:0/2489462063 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60ec0095f0 con 0x7f60fc0f5410 2023-12-25T21:10:24.450 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.449+0000 7f6100941700 1 -- 172.21.15.174:0/2489462063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60fc045ad0 con 0x7f60fc0f5410 2023-12-25T21:10:24.451 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.450+0000 7f60eaffd700 1 -- 172.21.15.174:0/2489462063 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f60ec01db70 con 0x7f60fc0f5410 2023-12-25T21:10:24.451 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:24.451 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:24.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.452+0000 7f6100941700 1 -- 172.21.15.174:0/2489462063 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f60e402ff60 msgr2=0x7f60e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:24.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.452+0000 7f6100941700 1 --2- 172.21.15.174:0/2489462063 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f60e402ff60 0x7f60e4032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f60f00096a0 tx=0x7f60f0006b40).stop 2023-12-25T21:10:24.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.452+0000 7f6100941700 1 -- 172.21.15.174:0/2489462063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60fc0f5410 msgr2=0x7f60fc0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:24.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.452+0000 7f6100941700 1 --2- 172.21.15.174:0/2489462063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60fc0f5410 0x7f60fc0fd3a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f60ec013510 tx=0x7f60ec00b040).stop 2023-12-25T21:10:24.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.452+0000 7f6100941700 1 -- 172.21.15.174:0/2489462063 shutdown_connections 2023-12-25T21:10:24.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.452+0000 7f6100941700 1 --2- 172.21.15.174:0/2489462063 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f60e402ff60 0x7f60e4032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:24.454 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.452+0000 7f6100941700 1 --2- 172.21.15.174:0/2489462063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60fc0f5410 0x7f60fc0fd3a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:24.454 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.452+0000 7f6100941700 1 -- 172.21.15.174:0/2489462063 >> 172.21.15.174:0/2489462063 conn(0x7f60fc0f0b70 msgr2=0x7f60fc05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:24.454 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.452+0000 7f6100941700 1 -- 172.21.15.174:0/2489462063 shutdown_connections 2023-12-25T21:10:24.454 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:24.452+0000 7f6100941700 1 -- 172.21.15.174:0/2489462063 wait complete. 2023-12-25T21:10:24.454 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:24.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:24 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:22.599427+0000 mgr.smithi061.jxxeke (mgr.14164) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:25.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:25 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:24.451631+0000 mon.smithi061 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.174:0/2489462063' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:25.917 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:25.917 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:26.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:26 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:24.599756+0000 mgr.smithi061.jxxeke (mgr.14164) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:27.075 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.074+0000 7f6f3f5c7700 1 -- 172.21.15.174:0/181379958 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f380f5410 msgr2=0x7f6f380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:27.076 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.074+0000 7f6f3f5c7700 1 --2- 172.21.15.174:0/181379958 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f380f5410 0x7f6f380f5830 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f6f28009fb0 tx=0x7f6f28009b30).stop 2023-12-25T21:10:27.076 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.074+0000 7f6f3f5c7700 1 -- 172.21.15.174:0/181379958 shutdown_connections 2023-12-25T21:10:27.076 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.074+0000 7f6f3f5c7700 1 --2- 172.21.15.174:0/181379958 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f380f5410 0x7f6f380f5830 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:27.076 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.074+0000 7f6f3f5c7700 1 -- 172.21.15.174:0/181379958 >> 172.21.15.174:0/181379958 conn(0x7f6f380f0b70 msgr2=0x7f6f380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:27.076 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.074+0000 7f6f3f5c7700 1 -- 172.21.15.174:0/181379958 shutdown_connections 2023-12-25T21:10:27.076 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.074+0000 7f6f3f5c7700 1 -- 172.21.15.174:0/181379958 wait complete. 2023-12-25T21:10:27.076 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.075+0000 7f6f3f5c7700 1 Processor -- start 2023-12-25T21:10:27.076 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.075+0000 7f6f3f5c7700 1 -- start start 2023-12-25T21:10:27.078 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.075+0000 7f6f3f5c7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f380f5410 0x7f6f38105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:27.078 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.075+0000 7f6f3f5c7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f38105b20 con 0x7f6f380f5410 2023-12-25T21:10:27.078 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.075+0000 7f6f3d363700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f380f5410 0x7f6f38105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:27.078 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.075+0000 7f6f3d363700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f380f5410 0x7f6f38105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:44814/0 (socket says 172.21.15.174:44814) 2023-12-25T21:10:27.078 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.075+0000 7f6f3d363700 1 -- 172.21.15.174:0/2202968373 learned_addr learned my addr 172.21.15.174:0/2202968373 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:27.078 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.076+0000 7f6f3d363700 1 -- 172.21.15.174:0/2202968373 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f28005040 con 0x7f6f380f5410 2023-12-25T21:10:27.078 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.076+0000 7f6f3d363700 1 --2- 172.21.15.174:0/2202968373 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f380f5410 0x7f6f38105630 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f6f28013510 tx=0x7f6f2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:27.078 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.076+0000 7f6f2dffb700 1 -- 172.21.15.174:0/2202968373 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f28009320 con 0x7f6f380f5410 2023-12-25T21:10:27.078 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.076+0000 7f6f3f5c7700 1 -- 172.21.15.174:0/2202968373 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f38105d20 con 0x7f6f380f5410 2023-12-25T21:10:27.078 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.076+0000 7f6f2dffb700 1 -- 172.21.15.174:0/2202968373 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f28013f00 con 0x7f6f380f5410 2023-12-25T21:10:27.078 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.076+0000 7f6f2dffb700 1 -- 172.21.15.174:0/2202968373 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f28014d90 con 0x7f6f380f5410 2023-12-25T21:10:27.079 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.076+0000 7f6f3f5c7700 1 -- 172.21.15.174:0/2202968373 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f381061b0 con 0x7f6f380f5410 2023-12-25T21:10:27.079 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.077+0000 7f6f2dffb700 1 -- 172.21.15.174:0/2202968373 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f28014420 con 0x7f6f380f5410 2023-12-25T21:10:27.079 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.077+0000 7f6f3f5c7700 1 -- 172.21.15.174:0/2202968373 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f38045ad0 con 0x7f6f380f5410 2023-12-25T21:10:27.079 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.077+0000 7f6f2dffb700 1 --2- 172.21.15.174:0/2202968373 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6f2402ffb0 0x7f6f24032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:27.079 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.077+0000 7f6f2dffb700 1 -- 172.21.15.174:0/2202968373 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f280410c0 con 0x7f6f380f5410 2023-12-25T21:10:27.079 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.077+0000 7f6f3cb62700 1 --2- 172.21.15.174:0/2202968373 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6f2402ffb0 0x7f6f24032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:27.079 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.078+0000 7f6f3cb62700 1 --2- 172.21.15.174:0/2202968373 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6f2402ffb0 0x7f6f24032470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f6f340096a0 tx=0x7f6f34006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:27.082 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.082+0000 7f6f2dffb700 1 -- 172.21.15.174:0/2202968373 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f280095f0 con 0x7f6f380f5410 2023-12-25T21:10:27.291 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.290+0000 7f6f3f5c7700 1 -- 172.21.15.174:0/2202968373 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f380fae60 con 0x7f6f380f5410 2023-12-25T21:10:27.291 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.291+0000 7f6f2dffb700 1 -- 172.21.15.174:0/2202968373 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6f2801db70 con 0x7f6f380f5410 2023-12-25T21:10:27.292 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:27.292 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:27.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.292+0000 7f6f3f5c7700 1 -- 172.21.15.174:0/2202968373 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6f2402ffb0 msgr2=0x7f6f24032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:27.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.292+0000 7f6f3f5c7700 1 --2- 172.21.15.174:0/2202968373 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6f2402ffb0 0x7f6f24032470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f6f340096a0 tx=0x7f6f34006b40).stop 2023-12-25T21:10:27.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.292+0000 7f6f3f5c7700 1 -- 172.21.15.174:0/2202968373 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f380f5410 msgr2=0x7f6f38105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:27.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.292+0000 7f6f3f5c7700 1 --2- 172.21.15.174:0/2202968373 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f380f5410 0x7f6f38105630 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f6f28013510 tx=0x7f6f2800b040).stop 2023-12-25T21:10:27.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.293+0000 7f6f3f5c7700 1 -- 172.21.15.174:0/2202968373 shutdown_connections 2023-12-25T21:10:27.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.293+0000 7f6f3f5c7700 1 --2- 172.21.15.174:0/2202968373 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6f2402ffb0 0x7f6f24032470 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:27.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.293+0000 7f6f3f5c7700 1 --2- 172.21.15.174:0/2202968373 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f380f5410 0x7f6f38105630 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:27.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.293+0000 7f6f3f5c7700 1 -- 172.21.15.174:0/2202968373 >> 172.21.15.174:0/2202968373 conn(0x7f6f380f0b70 msgr2=0x7f6f380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:27.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.293+0000 7f6f3f5c7700 1 -- 172.21.15.174:0/2202968373 shutdown_connections 2023-12-25T21:10:27.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:27.293+0000 7f6f3f5c7700 1 -- 172.21.15.174:0/2202968373 wait complete. 2023-12-25T21:10:27.295 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:27.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:27 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:27.292297+0000 mon.smithi061 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.174:0/2202968373' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:28.673 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:28.673 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:28.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:28 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:26.600183+0000 mgr.smithi061.jxxeke (mgr.14164) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:29.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:29 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:28.600549+0000 mgr.smithi061.jxxeke (mgr.14164) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:29.817 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.814+0000 7fa48fca4700 1 -- 172.21.15.174:0/3485798817 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4880f5410 msgr2=0x7fa4880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:29.817 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.814+0000 7fa48fca4700 1 --2- 172.21.15.174:0/3485798817 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4880f5410 0x7fa4880f5830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fa478009fb0 tx=0x7fa478009b30).stop 2023-12-25T21:10:29.817 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.814+0000 7fa48fca4700 1 -- 172.21.15.174:0/3485798817 shutdown_connections 2023-12-25T21:10:29.817 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.814+0000 7fa48fca4700 1 --2- 172.21.15.174:0/3485798817 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4880f5410 0x7fa4880f5830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:29.817 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.814+0000 7fa48fca4700 1 -- 172.21.15.174:0/3485798817 >> 172.21.15.174:0/3485798817 conn(0x7fa4880f0b70 msgr2=0x7fa4880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:29.817 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.815+0000 7fa48fca4700 1 -- 172.21.15.174:0/3485798817 shutdown_connections 2023-12-25T21:10:29.817 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.815+0000 7fa48fca4700 1 -- 172.21.15.174:0/3485798817 wait complete. 2023-12-25T21:10:29.817 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.815+0000 7fa48fca4700 1 Processor -- start 2023-12-25T21:10:29.817 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.815+0000 7fa48fca4700 1 -- start start 2023-12-25T21:10:29.817 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.815+0000 7fa48fca4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4880f5410 0x7fa488105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:29.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.815+0000 7fa48fca4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa488105b20 con 0x7fa4880f5410 2023-12-25T21:10:29.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.815+0000 7fa48da40700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4880f5410 0x7fa488105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:29.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.816+0000 7fa48da40700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4880f5410 0x7fa488105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:44832/0 (socket says 172.21.15.174:44832) 2023-12-25T21:10:29.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.816+0000 7fa48da40700 1 -- 172.21.15.174:0/2012435736 learned_addr learned my addr 172.21.15.174:0/2012435736 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:29.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.816+0000 7fa48da40700 1 -- 172.21.15.174:0/2012435736 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa478005040 con 0x7fa4880f5410 2023-12-25T21:10:29.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.816+0000 7fa48da40700 1 --2- 172.21.15.174:0/2012435736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4880f5410 0x7fa488105630 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fa478014000 tx=0x7fa47800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:29.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.816+0000 7fa47e7fc700 1 -- 172.21.15.174:0/2012435736 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4780095f0 con 0x7fa4880f5410 2023-12-25T21:10:29.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.816+0000 7fa47e7fc700 1 -- 172.21.15.174:0/2012435736 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa478007e70 con 0x7fa4880f5410 2023-12-25T21:10:29.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.816+0000 7fa48fca4700 1 -- 172.21.15.174:0/2012435736 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa488105d20 con 0x7fa4880f5410 2023-12-25T21:10:29.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.816+0000 7fa47e7fc700 1 -- 172.21.15.174:0/2012435736 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa478014ea0 con 0x7fa4880f5410 2023-12-25T21:10:29.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.816+0000 7fa48fca4700 1 -- 172.21.15.174:0/2012435736 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4881061b0 con 0x7fa4880f5410 2023-12-25T21:10:29.819 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.817+0000 7fa47e7fc700 1 -- 172.21.15.174:0/2012435736 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa478025070 con 0x7fa4880f5410 2023-12-25T21:10:29.819 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.817+0000 7fa48fca4700 1 -- 172.21.15.174:0/2012435736 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa488045ad0 con 0x7fa4880f5410 2023-12-25T21:10:29.819 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.817+0000 7fa47e7fc700 1 --2- 172.21.15.174:0/2012435736 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa47402ffb0 0x7fa474032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:29.819 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.817+0000 7fa47e7fc700 1 -- 172.21.15.174:0/2012435736 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa478040a30 con 0x7fa4880f5410 2023-12-25T21:10:29.819 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.817+0000 7fa48d23f700 1 --2- 172.21.15.174:0/2012435736 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa47402ffb0 0x7fa474032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:29.819 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.818+0000 7fa48d23f700 1 --2- 172.21.15.174:0/2012435736 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa47402ffb0 0x7fa474032470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fa4840096a0 tx=0x7fa484006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:29.822 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:29.822+0000 7fa47e7fc700 1 -- 172.21.15.174:0/2012435736 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa4780079c0 con 0x7fa4880f5410 2023-12-25T21:10:30.028 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:30.027+0000 7fa48fca4700 1 -- 172.21.15.174:0/2012435736 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4880fae60 con 0x7fa4880f5410 2023-12-25T21:10:30.029 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:30.028+0000 7fa47e7fc700 1 -- 172.21.15.174:0/2012435736 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa47800f300 con 0x7fa4880f5410 2023-12-25T21:10:30.029 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:30.029 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:30.030 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:30.029+0000 7fa48fca4700 1 -- 172.21.15.174:0/2012435736 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa47402ffb0 msgr2=0x7fa474032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:30.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:30.029+0000 7fa48fca4700 1 --2- 172.21.15.174:0/2012435736 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa47402ffb0 0x7fa474032470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fa4840096a0 tx=0x7fa484006b40).stop 2023-12-25T21:10:30.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:30.029+0000 7fa48fca4700 1 -- 172.21.15.174:0/2012435736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4880f5410 msgr2=0x7fa488105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:30.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:30.029+0000 7fa48fca4700 1 --2- 172.21.15.174:0/2012435736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4880f5410 0x7fa488105630 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fa478014000 tx=0x7fa47800b040).stop 2023-12-25T21:10:30.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:30.030+0000 7fa48fca4700 1 -- 172.21.15.174:0/2012435736 shutdown_connections 2023-12-25T21:10:30.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:30.030+0000 7fa48fca4700 1 --2- 172.21.15.174:0/2012435736 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa47402ffb0 0x7fa474032470 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:30.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:30.030+0000 7fa48fca4700 1 --2- 172.21.15.174:0/2012435736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4880f5410 0x7fa488105630 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:30.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:30.030+0000 7fa48fca4700 1 -- 172.21.15.174:0/2012435736 >> 172.21.15.174:0/2012435736 conn(0x7fa4880f0b70 msgr2=0x7fa4880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:30.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:30.030+0000 7fa48fca4700 1 -- 172.21.15.174:0/2012435736 shutdown_connections 2023-12-25T21:10:30.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:30.030+0000 7fa48fca4700 1 -- 172.21.15.174:0/2012435736 wait complete. 2023-12-25T21:10:30.031 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:30.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:30 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:30.029449+0000 mon.smithi061 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.174:0/2012435736' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:31.443 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:31.443 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:32.573 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.571+0000 7f1b79e70700 1 -- 172.21.15.174:0/491593711 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b740f5410 msgr2=0x7f1b740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:32.573 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.571+0000 7f1b79e70700 1 --2- 172.21.15.174:0/491593711 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b740f5410 0x7f1b740f5830 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f1b68009fb0 tx=0x7f1b68009b30).stop 2023-12-25T21:10:32.573 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.571+0000 7f1b79e70700 1 -- 172.21.15.174:0/491593711 shutdown_connections 2023-12-25T21:10:32.573 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.571+0000 7f1b79e70700 1 --2- 172.21.15.174:0/491593711 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b740f5410 0x7f1b740f5830 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:32.573 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.571+0000 7f1b79e70700 1 -- 172.21.15.174:0/491593711 >> 172.21.15.174:0/491593711 conn(0x7f1b740f0b70 msgr2=0x7f1b740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:32.573 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.571+0000 7f1b79e70700 1 -- 172.21.15.174:0/491593711 shutdown_connections 2023-12-25T21:10:32.573 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.571+0000 7f1b79e70700 1 -- 172.21.15.174:0/491593711 wait complete. 2023-12-25T21:10:32.573 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.572+0000 7f1b79e70700 1 Processor -- start 2023-12-25T21:10:32.573 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.572+0000 7f1b79e70700 1 -- start start 2023-12-25T21:10:32.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.572+0000 7f1b79e70700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b740f5410 0x7f1b7410b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:32.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.572+0000 7f1b79e70700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b74107240 con 0x7f1b740f5410 2023-12-25T21:10:32.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.572+0000 7f1b737fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b740f5410 0x7f1b7410b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:32.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.572+0000 7f1b737fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b740f5410 0x7f1b7410b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:44858/0 (socket says 172.21.15.174:44858) 2023-12-25T21:10:32.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.572+0000 7f1b737fe700 1 -- 172.21.15.174:0/1360649356 learned_addr learned my addr 172.21.15.174:0/1360649356 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:32.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.572+0000 7f1b737fe700 1 -- 172.21.15.174:0/1360649356 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b68005040 con 0x7f1b740f5410 2023-12-25T21:10:32.575 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.573+0000 7f1b737fe700 1 --2- 172.21.15.174:0/1360649356 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b740f5410 0x7f1b7410b850 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f1b68014000 tx=0x7f1b6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:32.575 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.573+0000 7f1b5bfff700 1 -- 172.21.15.174:0/1360649356 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b680095f0 con 0x7f1b740f5410 2023-12-25T21:10:32.575 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.573+0000 7f1b79e70700 1 -- 172.21.15.174:0/1360649356 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b74060220 con 0x7f1b740f5410 2023-12-25T21:10:32.575 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.573+0000 7f1b5bfff700 1 -- 172.21.15.174:0/1360649356 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b68007e70 con 0x7f1b740f5410 2023-12-25T21:10:32.575 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.573+0000 7f1b5bfff700 1 -- 172.21.15.174:0/1360649356 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b68014ea0 con 0x7f1b740f5410 2023-12-25T21:10:32.575 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.573+0000 7f1b79e70700 1 -- 172.21.15.174:0/1360649356 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7410bd40 con 0x7f1b740f5410 2023-12-25T21:10:32.575 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.573+0000 7f1b5bfff700 1 -- 172.21.15.174:0/1360649356 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b68025070 con 0x7f1b740f5410 2023-12-25T21:10:32.575 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.574+0000 7f1b5bfff700 1 --2- 172.21.15.174:0/1360649356 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1b5c02ff60 0x7f1b5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:32.576 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.574+0000 7f1b79e70700 1 -- 172.21.15.174:0/1360649356 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b74045ad0 con 0x7f1b740f5410 2023-12-25T21:10:32.576 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.574+0000 7f1b5bfff700 1 -- 172.21.15.174:0/1360649356 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b68040d50 con 0x7f1b740f5410 2023-12-25T21:10:32.576 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.574+0000 7f1b72ffd700 1 --2- 172.21.15.174:0/1360649356 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1b5c02ff60 0x7f1b5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:32.576 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.574+0000 7f1b72ffd700 1 --2- 172.21.15.174:0/1360649356 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1b5c02ff60 0x7f1b5c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f1b640096a0 tx=0x7f1b64006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:32.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.579+0000 7f1b5bfff700 1 -- 172.21.15.174:0/1360649356 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b68026350 con 0x7f1b740f5410 2023-12-25T21:10:32.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:32 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:30.600923+0000 mgr.smithi061.jxxeke (mgr.14164) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:32.785 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.784+0000 7f1b79e70700 1 -- 172.21.15.174:0/1360649356 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b740faf50 con 0x7f1b740f5410 2023-12-25T21:10:32.786 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.785+0000 7f1b5bfff700 1 -- 172.21.15.174:0/1360649356 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1b68025770 con 0x7f1b740f5410 2023-12-25T21:10:32.786 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:32.786 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:32.788 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.787+0000 7f1b79e70700 1 -- 172.21.15.174:0/1360649356 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1b5c02ff60 msgr2=0x7f1b5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:32.788 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.787+0000 7f1b79e70700 1 --2- 172.21.15.174:0/1360649356 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1b5c02ff60 0x7f1b5c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f1b640096a0 tx=0x7f1b64006b40).stop 2023-12-25T21:10:32.788 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.787+0000 7f1b79e70700 1 -- 172.21.15.174:0/1360649356 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b740f5410 msgr2=0x7f1b7410b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:32.788 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.787+0000 7f1b79e70700 1 --2- 172.21.15.174:0/1360649356 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b740f5410 0x7f1b7410b850 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f1b68014000 tx=0x7f1b6800b040).stop 2023-12-25T21:10:32.788 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.787+0000 7f1b79e70700 1 -- 172.21.15.174:0/1360649356 shutdown_connections 2023-12-25T21:10:32.789 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.787+0000 7f1b79e70700 1 --2- 172.21.15.174:0/1360649356 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f1b5c02ff60 0x7f1b5c032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:32.789 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.787+0000 7f1b79e70700 1 --2- 172.21.15.174:0/1360649356 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b740f5410 0x7f1b7410b850 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:32.789 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.787+0000 7f1b79e70700 1 -- 172.21.15.174:0/1360649356 >> 172.21.15.174:0/1360649356 conn(0x7f1b740f0b70 msgr2=0x7f1b740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:32.789 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.787+0000 7f1b79e70700 1 -- 172.21.15.174:0/1360649356 shutdown_connections 2023-12-25T21:10:32.789 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:32.787+0000 7f1b79e70700 1 -- 172.21.15.174:0/1360649356 wait complete. 2023-12-25T21:10:32.789 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:33.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:33 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:32.786499+0000 mon.smithi061 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.174:0/1360649356' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:34.172 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:34.172 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:34.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:34 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:32.601147+0000 mgr.smithi061.jxxeke (mgr.14164) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:35.291 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.290+0000 7f258f3ba700 1 -- 172.21.15.174:0/914298353 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25880f51a0 msgr2=0x7f25880f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:35.291 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.290+0000 7f258f3ba700 1 --2- 172.21.15.174:0/914298353 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25880f51a0 0x7f25880f55c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f2578009fb0 tx=0x7f2578009b30).stop 2023-12-25T21:10:35.291 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.290+0000 7f258f3ba700 1 -- 172.21.15.174:0/914298353 shutdown_connections 2023-12-25T21:10:35.291 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.290+0000 7f258f3ba700 1 --2- 172.21.15.174:0/914298353 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25880f51a0 0x7f25880f55c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:35.291 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.290+0000 7f258f3ba700 1 -- 172.21.15.174:0/914298353 >> 172.21.15.174:0/914298353 conn(0x7f25880f09a0 msgr2=0x7f25880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:35.291 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.290+0000 7f258f3ba700 1 -- 172.21.15.174:0/914298353 shutdown_connections 2023-12-25T21:10:35.292 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.290+0000 7f258f3ba700 1 -- 172.21.15.174:0/914298353 wait complete. 2023-12-25T21:10:35.292 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.290+0000 7f258f3ba700 1 Processor -- start 2023-12-25T21:10:35.292 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.290+0000 7f258f3ba700 1 -- start start 2023-12-25T21:10:35.292 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.290+0000 7f258f3ba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25880f51a0 0x7f25881091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:35.292 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.290+0000 7f258f3ba700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2588060240 con 0x7f25880f51a0 2023-12-25T21:10:35.292 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.291+0000 7f258e3b8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25880f51a0 0x7f25881091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:35.292 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.291+0000 7f258e3b8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25880f51a0 0x7f25881091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:44884/0 (socket says 172.21.15.174:44884) 2023-12-25T21:10:35.292 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.291+0000 7f258e3b8700 1 -- 172.21.15.174:0/1400106948 learned_addr learned my addr 172.21.15.174:0/1400106948 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:35.293 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.291+0000 7f258e3b8700 1 -- 172.21.15.174:0/1400106948 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2578005040 con 0x7f25880f51a0 2023-12-25T21:10:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.291+0000 7f258e3b8700 1 --2- 172.21.15.174:0/1400106948 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25880f51a0 0x7f25881091b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f2578009830 tx=0x7f257800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.291+0000 7f257effd700 1 -- 172.21.15.174:0/1400106948 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2578009320 con 0x7f25880f51a0 2023-12-25T21:10:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.291+0000 7f258f3ba700 1 -- 172.21.15.174:0/1400106948 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2588060440 con 0x7f25880f51a0 2023-12-25T21:10:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.291+0000 7f257effd700 1 -- 172.21.15.174:0/1400106948 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25780136c0 con 0x7f25880f51a0 2023-12-25T21:10:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.291+0000 7f257effd700 1 -- 172.21.15.174:0/1400106948 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f257801ee90 con 0x7f25880f51a0 2023-12-25T21:10:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.291+0000 7f258f3ba700 1 -- 172.21.15.174:0/1400106948 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2588109860 con 0x7f25880f51a0 2023-12-25T21:10:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.292+0000 7f257effd700 1 -- 172.21.15.174:0/1400106948 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f257801e420 con 0x7f25880f51a0 2023-12-25T21:10:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.292+0000 7f258f3ba700 1 -- 172.21.15.174:0/1400106948 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2588045ad0 con 0x7f25880f51a0 2023-12-25T21:10:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.292+0000 7f257effd700 1 --2- 172.21.15.174:0/1400106948 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f257402ff10 0x7f25740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.292+0000 7f257effd700 1 -- 172.21.15.174:0/1400106948 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2578040fa0 con 0x7f25880f51a0 2023-12-25T21:10:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.292+0000 7f258dbb7700 1 --2- 172.21.15.174:0/1400106948 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f257402ff10 0x7f25740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:35.294 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.293+0000 7f258dbb7700 1 --2- 172.21.15.174:0/1400106948 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f257402ff10 0x7f25740323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f25840096a0 tx=0x7f2584006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:35.297 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.297+0000 7f257effd700 1 -- 172.21.15.174:0/1400106948 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25780095f0 con 0x7f25880f51a0 2023-12-25T21:10:35.505 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.505+0000 7f258f3ba700 1 -- 172.21.15.174:0/1400106948 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f258810a080 con 0x7f25880f51a0 2023-12-25T21:10:35.506 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.505+0000 7f257effd700 1 -- 172.21.15.174:0/1400106948 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f257802c2d0 con 0x7f25880f51a0 2023-12-25T21:10:35.506 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:35.506 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:35.508 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.507+0000 7f258f3ba700 1 -- 172.21.15.174:0/1400106948 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f257402ff10 msgr2=0x7f25740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:35.508 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.507+0000 7f258f3ba700 1 --2- 172.21.15.174:0/1400106948 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f257402ff10 0x7f25740323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f25840096a0 tx=0x7f2584006b40).stop 2023-12-25T21:10:35.508 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.507+0000 7f258f3ba700 1 -- 172.21.15.174:0/1400106948 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25880f51a0 msgr2=0x7f25881091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:35.508 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.507+0000 7f258f3ba700 1 --2- 172.21.15.174:0/1400106948 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25880f51a0 0x7f25881091b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f2578009830 tx=0x7f257800b040).stop 2023-12-25T21:10:35.508 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.507+0000 7f258f3ba700 1 -- 172.21.15.174:0/1400106948 shutdown_connections 2023-12-25T21:10:35.508 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.507+0000 7f258f3ba700 1 --2- 172.21.15.174:0/1400106948 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f257402ff10 0x7f25740323d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:35.508 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.507+0000 7f258f3ba700 1 --2- 172.21.15.174:0/1400106948 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25880f51a0 0x7f25881091b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:35.508 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.507+0000 7f258f3ba700 1 -- 172.21.15.174:0/1400106948 >> 172.21.15.174:0/1400106948 conn(0x7f25880f09a0 msgr2=0x7f25880f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:35.508 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.507+0000 7f258f3ba700 1 -- 172.21.15.174:0/1400106948 shutdown_connections 2023-12-25T21:10:35.508 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:35.507+0000 7f258f3ba700 1 -- 172.21.15.174:0/1400106948 wait complete. 2023-12-25T21:10:35.509 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:36.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:36 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:34.601421+0000 mgr.smithi061.jxxeke (mgr.14164) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:36.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:36 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:35.506761+0000 mon.smithi061 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.174:0/1400106948' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:36.872 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:36.873 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:38.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.007+0000 7f29ce6ab700 1 -- 172.21.15.174:0/809358435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c80f6510 msgr2=0x7f29c80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:38.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.007+0000 7f29ce6ab700 1 --2- 172.21.15.174:0/809358435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c80f6510 0x7f29c80f6930 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f29bc009fb0 tx=0x7f29bc009b30).stop 2023-12-25T21:10:38.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.007+0000 7f29ce6ab700 1 -- 172.21.15.174:0/809358435 shutdown_connections 2023-12-25T21:10:38.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.007+0000 7f29ce6ab700 1 --2- 172.21.15.174:0/809358435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c80f6510 0x7f29c80f6930 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:38.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.007+0000 7f29ce6ab700 1 -- 172.21.15.174:0/809358435 >> 172.21.15.174:0/809358435 conn(0x7f29c80f1c70 msgr2=0x7f29c80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:38.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.007+0000 7f29ce6ab700 1 -- 172.21.15.174:0/809358435 shutdown_connections 2023-12-25T21:10:38.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.007+0000 7f29ce6ab700 1 -- 172.21.15.174:0/809358435 wait complete. 2023-12-25T21:10:38.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.008+0000 7f29ce6ab700 1 Processor -- start 2023-12-25T21:10:38.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.008+0000 7f29ce6ab700 1 -- start start 2023-12-25T21:10:38.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.008+0000 7f29ce6ab700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c80f6510 0x7f29c8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:38.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.008+0000 7f29ce6ab700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c8106c20 con 0x7f29c80f6510 2023-12-25T21:10:38.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.008+0000 7f29c7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c80f6510 0x7f29c8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:38.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.008+0000 7f29c7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c80f6510 0x7f29c8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:51100/0 (socket says 172.21.15.174:51100) 2023-12-25T21:10:38.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.008+0000 7f29c7fff700 1 -- 172.21.15.174:0/1059546180 learned_addr learned my addr 172.21.15.174:0/1059546180 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:38.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.008+0000 7f29c7fff700 1 -- 172.21.15.174:0/1059546180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29bc005040 con 0x7f29c80f6510 2023-12-25T21:10:38.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.009+0000 7f29c7fff700 1 --2- 172.21.15.174:0/1059546180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c80f6510 0x7f29c8106730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f29bc014000 tx=0x7f29bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:38.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.009+0000 7f29c4ff9700 1 -- 172.21.15.174:0/1059546180 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29bc0095f0 con 0x7f29c80f6510 2023-12-25T21:10:38.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.009+0000 7f29ce6ab700 1 -- 172.21.15.174:0/1059546180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c8106e20 con 0x7f29c80f6510 2023-12-25T21:10:38.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.009+0000 7f29c4ff9700 1 -- 172.21.15.174:0/1059546180 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29bc007e70 con 0x7f29c80f6510 2023-12-25T21:10:38.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.009+0000 7f29c4ff9700 1 -- 172.21.15.174:0/1059546180 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29bc014ea0 con 0x7f29c80f6510 2023-12-25T21:10:38.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.009+0000 7f29ce6ab700 1 -- 172.21.15.174:0/1059546180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c81072b0 con 0x7f29c80f6510 2023-12-25T21:10:38.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.010+0000 7f29c4ff9700 1 -- 172.21.15.174:0/1059546180 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29bc025070 con 0x7f29c80f6510 2023-12-25T21:10:38.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.010+0000 7f29ce6ab700 1 -- 172.21.15.174:0/1059546180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c8045ad0 con 0x7f29c80f6510 2023-12-25T21:10:38.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.010+0000 7f29c4ff9700 1 --2- 172.21.15.174:0/1059546180 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f29b002ff60 0x7f29b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:38.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.010+0000 7f29c4ff9700 1 -- 172.21.15.174:0/1059546180 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f29bc040c60 con 0x7f29c80f6510 2023-12-25T21:10:38.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.010+0000 7f29c77fe700 1 --2- 172.21.15.174:0/1059546180 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f29b002ff60 0x7f29b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:38.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.010+0000 7f29c77fe700 1 --2- 172.21.15.174:0/1059546180 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f29b002ff60 0x7f29b0032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f29b80096a0 tx=0x7f29b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:38.015 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.015+0000 7f29c4ff9700 1 -- 172.21.15.174:0/1059546180 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29bc025950 con 0x7f29c80f6510 2023-12-25T21:10:38.219 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.218+0000 7f29ce6ab700 1 -- 172.21.15.174:0/1059546180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29c80fbf60 con 0x7f29c80f6510 2023-12-25T21:10:38.220 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.219+0000 7f29c4ff9700 1 -- 172.21.15.174:0/1059546180 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f29bc025950 con 0x7f29c80f6510 2023-12-25T21:10:38.220 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:38.220 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:38.221 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.220+0000 7f29ce6ab700 1 -- 172.21.15.174:0/1059546180 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f29b002ff60 msgr2=0x7f29b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:38.221 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.220+0000 7f29ce6ab700 1 --2- 172.21.15.174:0/1059546180 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f29b002ff60 0x7f29b0032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f29b80096a0 tx=0x7f29b8006b40).stop 2023-12-25T21:10:38.222 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.220+0000 7f29ce6ab700 1 -- 172.21.15.174:0/1059546180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c80f6510 msgr2=0x7f29c8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:38.222 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.220+0000 7f29ce6ab700 1 --2- 172.21.15.174:0/1059546180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c80f6510 0x7f29c8106730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f29bc014000 tx=0x7f29bc00b040).stop 2023-12-25T21:10:38.222 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.221+0000 7f29ce6ab700 1 -- 172.21.15.174:0/1059546180 shutdown_connections 2023-12-25T21:10:38.222 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.221+0000 7f29ce6ab700 1 --2- 172.21.15.174:0/1059546180 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f29b002ff60 0x7f29b0032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:38.222 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.221+0000 7f29ce6ab700 1 --2- 172.21.15.174:0/1059546180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c80f6510 0x7f29c8106730 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:38.222 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.221+0000 7f29ce6ab700 1 -- 172.21.15.174:0/1059546180 >> 172.21.15.174:0/1059546180 conn(0x7f29c80f1c70 msgr2=0x7f29c80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:38.222 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.221+0000 7f29ce6ab700 1 -- 172.21.15.174:0/1059546180 shutdown_connections 2023-12-25T21:10:38.222 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:38.221+0000 7f29ce6ab700 1 -- 172.21.15.174:0/1059546180 wait complete. 2023-12-25T21:10:38.222 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:38.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:38 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:36.601765+0000 mgr.smithi061.jxxeke (mgr.14164) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:38.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:38 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:38.220385+0000 mon.smithi061 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.174:0/1059546180' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:39.621 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:39.621 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:40.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:40 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:38.602077+0000 mgr.smithi061.jxxeke (mgr.14164) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:40.847 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.846+0000 7f116510a700 1 -- 172.21.15.174:0/2037564222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1160060220 msgr2=0x7f11600f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:40.847 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.846+0000 7f116510a700 1 --2- 172.21.15.174:0/2037564222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1160060220 0x7f11600f9150 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f1148009fb0 tx=0x7f1148009b30).stop 2023-12-25T21:10:40.847 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.846+0000 7f116510a700 1 -- 172.21.15.174:0/2037564222 shutdown_connections 2023-12-25T21:10:40.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.846+0000 7f116510a700 1 --2- 172.21.15.174:0/2037564222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1160060220 0x7f11600f9150 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:40.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.846+0000 7f116510a700 1 -- 172.21.15.174:0/2037564222 >> 172.21.15.174:0/2037564222 conn(0x7f11600f0b70 msgr2=0x7f11600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:40.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.846+0000 7f116510a700 1 -- 172.21.15.174:0/2037564222 shutdown_connections 2023-12-25T21:10:40.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.846+0000 7f116510a700 1 -- 172.21.15.174:0/2037564222 wait complete. 2023-12-25T21:10:40.850 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.847+0000 7f116510a700 1 Processor -- start 2023-12-25T21:10:40.850 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.847+0000 7f116510a700 1 -- start start 2023-12-25T21:10:40.850 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.847+0000 7f116510a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1160060220 0x7f11601066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:40.850 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.847+0000 7f116510a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1160106bd0 con 0x7f1160060220 2023-12-25T21:10:40.850 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.847+0000 7f115ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1160060220 0x7f11601066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:40.850 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.847+0000 7f115ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1160060220 0x7f11601066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:51126/0 (socket says 172.21.15.174:51126) 2023-12-25T21:10:40.850 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.847+0000 7f115ed9d700 1 -- 172.21.15.174:0/1935589272 learned_addr learned my addr 172.21.15.174:0/1935589272 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:40.850 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.847+0000 7f115ed9d700 1 -- 172.21.15.174:0/1935589272 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1148005040 con 0x7f1160060220 2023-12-25T21:10:40.850 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.848+0000 7f115ed9d700 1 --2- 172.21.15.174:0/1935589272 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1160060220 0x7f11601066e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f1148013510 tx=0x7f114800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:40.850 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.848+0000 7f11577fe700 1 -- 172.21.15.174:0/1935589272 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1148009320 con 0x7f1160060220 2023-12-25T21:10:40.851 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.848+0000 7f116510a700 1 -- 172.21.15.174:0/1935589272 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1160106dd0 con 0x7f1160060220 2023-12-25T21:10:40.851 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.848+0000 7f11577fe700 1 -- 172.21.15.174:0/1935589272 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1148013f00 con 0x7f1160060220 2023-12-25T21:10:40.851 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.848+0000 7f11577fe700 1 -- 172.21.15.174:0/1935589272 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1148014dc0 con 0x7f1160060220 2023-12-25T21:10:40.851 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.848+0000 7f116510a700 1 -- 172.21.15.174:0/1935589272 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1160107260 con 0x7f1160060220 2023-12-25T21:10:40.851 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.849+0000 7f116510a700 1 -- 172.21.15.174:0/1935589272 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1160045ad0 con 0x7f1160060220 2023-12-25T21:10:40.851 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.849+0000 7f11577fe700 1 -- 172.21.15.174:0/1935589272 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f11480095f0 con 0x7f1160060220 2023-12-25T21:10:40.851 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.849+0000 7f11577fe700 1 --2- 172.21.15.174:0/1935589272 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f114c02ff60 0x7f114c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:40.851 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.849+0000 7f11577fe700 1 -- 172.21.15.174:0/1935589272 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1148041000 con 0x7f1160060220 2023-12-25T21:10:40.851 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.849+0000 7f115e59c700 1 --2- 172.21.15.174:0/1935589272 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f114c02ff60 0x7f114c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:40.851 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.849+0000 7f115e59c700 1 --2- 172.21.15.174:0/1935589272 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f114c02ff60 0x7f114c032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f11500096a0 tx=0x7f1150006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:40.854 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:40.854+0000 7f11577fe700 1 -- 172.21.15.174:0/1935589272 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f114801c790 con 0x7f1160060220 2023-12-25T21:10:41.056 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:41.055+0000 7f116510a700 1 -- 172.21.15.174:0/1935589272 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1160108110 con 0x7f1160060220 2023-12-25T21:10:41.057 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:41.056+0000 7f11577fe700 1 -- 172.21.15.174:0/1935589272 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1148010230 con 0x7f1160060220 2023-12-25T21:10:41.057 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:41.057 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:41.058 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:41.058+0000 7f116510a700 1 -- 172.21.15.174:0/1935589272 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f114c02ff60 msgr2=0x7f114c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:41.059 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:41.058+0000 7f116510a700 1 --2- 172.21.15.174:0/1935589272 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f114c02ff60 0x7f114c032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f11500096a0 tx=0x7f1150006b40).stop 2023-12-25T21:10:41.059 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:41.058+0000 7f116510a700 1 -- 172.21.15.174:0/1935589272 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1160060220 msgr2=0x7f11601066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:41.059 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:41.058+0000 7f116510a700 1 --2- 172.21.15.174:0/1935589272 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1160060220 0x7f11601066e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f1148013510 tx=0x7f114800b040).stop 2023-12-25T21:10:41.059 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:41.058+0000 7f116510a700 1 -- 172.21.15.174:0/1935589272 shutdown_connections 2023-12-25T21:10:41.059 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:41.058+0000 7f116510a700 1 --2- 172.21.15.174:0/1935589272 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f114c02ff60 0x7f114c032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:41.059 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:41.058+0000 7f116510a700 1 --2- 172.21.15.174:0/1935589272 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1160060220 0x7f11601066e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:41.059 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:41.058+0000 7f116510a700 1 -- 172.21.15.174:0/1935589272 >> 172.21.15.174:0/1935589272 conn(0x7f11600f0b70 msgr2=0x7f11600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:41.059 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:41.058+0000 7f116510a700 1 -- 172.21.15.174:0/1935589272 shutdown_connections 2023-12-25T21:10:41.059 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:41.058+0000 7f116510a700 1 -- 172.21.15.174:0/1935589272 wait complete. 2023-12-25T21:10:41.060 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:41.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:41 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:41.057522+0000 mon.smithi061 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.174:0/1935589272' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:42.445 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:42.445 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:42.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:42 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:40.602333+0000 mgr.smithi061.jxxeke (mgr.14164) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:43.627 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.624+0000 7fbfeee5d700 1 -- 172.21.15.174:0/1491978497 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe80f6480 msgr2=0x7fbfe80f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:43.627 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.624+0000 7fbfeee5d700 1 --2- 172.21.15.174:0/1491978497 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe80f6480 0x7fbfe80f68a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fbfdc009fb0 tx=0x7fbfdc009b30).stop 2023-12-25T21:10:43.627 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.624+0000 7fbfeee5d700 1 -- 172.21.15.174:0/1491978497 shutdown_connections 2023-12-25T21:10:43.627 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.624+0000 7fbfeee5d700 1 --2- 172.21.15.174:0/1491978497 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe80f6480 0x7fbfe80f68a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:43.627 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.624+0000 7fbfeee5d700 1 -- 172.21.15.174:0/1491978497 >> 172.21.15.174:0/1491978497 conn(0x7fbfe80f1c00 msgr2=0x7fbfe80f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:43.627 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.624+0000 7fbfeee5d700 1 -- 172.21.15.174:0/1491978497 shutdown_connections 2023-12-25T21:10:43.627 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.625+0000 7fbfeee5d700 1 -- 172.21.15.174:0/1491978497 wait complete. 2023-12-25T21:10:43.627 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.625+0000 7fbfeee5d700 1 Processor -- start 2023-12-25T21:10:43.627 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.625+0000 7fbfeee5d700 1 -- start start 2023-12-25T21:10:43.627 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.625+0000 7fbfeee5d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe80f6480 0x7fbfe81066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:43.628 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.625+0000 7fbfeee5d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe8106bc0 con 0x7fbfe80f6480 2023-12-25T21:10:43.628 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.626+0000 7fbfecbf9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe80f6480 0x7fbfe81066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:43.628 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.626+0000 7fbfecbf9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe80f6480 0x7fbfe81066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:51138/0 (socket says 172.21.15.174:51138) 2023-12-25T21:10:43.628 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.626+0000 7fbfecbf9700 1 -- 172.21.15.174:0/334405738 learned_addr learned my addr 172.21.15.174:0/334405738 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:43.628 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.626+0000 7fbfecbf9700 1 -- 172.21.15.174:0/334405738 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc005040 con 0x7fbfe80f6480 2023-12-25T21:10:43.628 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.626+0000 7fbfecbf9700 1 --2- 172.21.15.174:0/334405738 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe80f6480 0x7fbfe81066d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fbfdc013510 tx=0x7fbfdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:43.628 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.626+0000 7fbfe57fa700 1 -- 172.21.15.174:0/334405738 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfdc009320 con 0x7fbfe80f6480 2023-12-25T21:10:43.628 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.626+0000 7fbfeee5d700 1 -- 172.21.15.174:0/334405738 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe8106dc0 con 0x7fbfe80f6480 2023-12-25T21:10:43.628 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.626+0000 7fbfe57fa700 1 -- 172.21.15.174:0/334405738 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbfdc013f00 con 0x7fbfe80f6480 2023-12-25T21:10:43.628 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.626+0000 7fbfe57fa700 1 -- 172.21.15.174:0/334405738 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfdc014d90 con 0x7fbfe80f6480 2023-12-25T21:10:43.628 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.626+0000 7fbfeee5d700 1 -- 172.21.15.174:0/334405738 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe8107250 con 0x7fbfe80f6480 2023-12-25T21:10:43.629 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.627+0000 7fbfe57fa700 1 -- 172.21.15.174:0/334405738 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbfdc014420 con 0x7fbfe80f6480 2023-12-25T21:10:43.629 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.627+0000 7fbfeee5d700 1 -- 172.21.15.174:0/334405738 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe8045ad0 con 0x7fbfe80f6480 2023-12-25T21:10:43.629 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.627+0000 7fbfe57fa700 1 --2- 172.21.15.174:0/334405738 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbfd802ffb0 0x7fbfd8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:43.629 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.627+0000 7fbfe57fa700 1 -- 172.21.15.174:0/334405738 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbfdc041070 con 0x7fbfe80f6480 2023-12-25T21:10:43.629 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.627+0000 7fbfe7fff700 1 --2- 172.21.15.174:0/334405738 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbfd802ffb0 0x7fbfd8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:43.629 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.628+0000 7fbfe7fff700 1 --2- 172.21.15.174:0/334405738 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbfd802ffb0 0x7fbfd8032470 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fbfd40096a0 tx=0x7fbfd4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:43.632 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.632+0000 7fbfe57fa700 1 -- 172.21.15.174:0/334405738 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbfdc06d2b0 con 0x7fbfe80f6480 2023-12-25T21:10:43.838 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.837+0000 7fbfeee5d700 1 -- 172.21.15.174:0/334405738 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbfe8108100 con 0x7fbfe80f6480 2023-12-25T21:10:43.838 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.838+0000 7fbfe57fa700 1 -- 172.21.15.174:0/334405738 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbfdc0095f0 con 0x7fbfe80f6480 2023-12-25T21:10:43.839 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:43.839 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:43.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.839+0000 7fbfeee5d700 1 -- 172.21.15.174:0/334405738 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbfd802ffb0 msgr2=0x7fbfd8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:43.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.839+0000 7fbfeee5d700 1 --2- 172.21.15.174:0/334405738 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbfd802ffb0 0x7fbfd8032470 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fbfd40096a0 tx=0x7fbfd4006b40).stop 2023-12-25T21:10:43.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.839+0000 7fbfeee5d700 1 -- 172.21.15.174:0/334405738 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe80f6480 msgr2=0x7fbfe81066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:43.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.839+0000 7fbfeee5d700 1 --2- 172.21.15.174:0/334405738 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe80f6480 0x7fbfe81066d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fbfdc013510 tx=0x7fbfdc00b040).stop 2023-12-25T21:10:43.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.839+0000 7fbfeee5d700 1 -- 172.21.15.174:0/334405738 shutdown_connections 2023-12-25T21:10:43.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.839+0000 7fbfeee5d700 1 --2- 172.21.15.174:0/334405738 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbfd802ffb0 0x7fbfd8032470 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:43.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.839+0000 7fbfeee5d700 1 --2- 172.21.15.174:0/334405738 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe80f6480 0x7fbfe81066d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:43.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.839+0000 7fbfeee5d700 1 -- 172.21.15.174:0/334405738 >> 172.21.15.174:0/334405738 conn(0x7fbfe80f1c00 msgr2=0x7fbfe80f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:43.841 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.839+0000 7fbfeee5d700 1 -- 172.21.15.174:0/334405738 shutdown_connections 2023-12-25T21:10:43.841 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:43.839+0000 7fbfeee5d700 1 -- 172.21.15.174:0/334405738 wait complete. 2023-12-25T21:10:43.841 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:44.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:44 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:42.602586+0000 mgr.smithi061.jxxeke (mgr.14164) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:44.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:44 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:43.839125+0000 mon.smithi061 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.174:0/334405738' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:45.229 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:45.230 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:46.406 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.403+0000 7fd913a67700 1 -- 172.21.15.174:0/2384770029 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90c0f47c0 msgr2=0x7fd90c0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:46.406 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.403+0000 7fd913a67700 1 --2- 172.21.15.174:0/2384770029 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90c0f47c0 0x7fd90c0f4be0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fd8fc009fb0 tx=0x7fd8fc009b30).stop 2023-12-25T21:10:46.406 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.403+0000 7fd913a67700 1 -- 172.21.15.174:0/2384770029 shutdown_connections 2023-12-25T21:10:46.406 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.403+0000 7fd913a67700 1 --2- 172.21.15.174:0/2384770029 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90c0f47c0 0x7fd90c0f4be0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:46.406 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.403+0000 7fd913a67700 1 -- 172.21.15.174:0/2384770029 >> 172.21.15.174:0/2384770029 conn(0x7fd90c0f02e0 msgr2=0x7fd90c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:46.406 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.403+0000 7fd913a67700 1 -- 172.21.15.174:0/2384770029 shutdown_connections 2023-12-25T21:10:46.406 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.403+0000 7fd913a67700 1 -- 172.21.15.174:0/2384770029 wait complete. 2023-12-25T21:10:46.406 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.404+0000 7fd913a67700 1 Processor -- start 2023-12-25T21:10:46.406 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.404+0000 7fd913a67700 1 -- start start 2023-12-25T21:10:46.406 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.404+0000 7fd913a67700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90c0f47c0 0x7fd90c0f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:46.406 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.404+0000 7fd913a67700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c0f57f0 con 0x7fd90c0f47c0 2023-12-25T21:10:46.407 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.404+0000 7fd911803700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90c0f47c0 0x7fd90c0f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:46.407 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.404+0000 7fd911803700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90c0f47c0 0x7fd90c0f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:36124/0 (socket says 172.21.15.174:36124) 2023-12-25T21:10:46.407 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.404+0000 7fd911803700 1 -- 172.21.15.174:0/1218805456 learned_addr learned my addr 172.21.15.174:0/1218805456 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:46.407 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.405+0000 7fd911803700 1 -- 172.21.15.174:0/1218805456 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc005040 con 0x7fd90c0f47c0 2023-12-25T21:10:46.407 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.405+0000 7fd911803700 1 --2- 172.21.15.174:0/1218805456 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90c0f47c0 0x7fd90c0f7370 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fd8fc014000 tx=0x7fd8fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:46.407 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.405+0000 7fd9027fc700 1 -- 172.21.15.174:0/1218805456 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8fc0095f0 con 0x7fd90c0f47c0 2023-12-25T21:10:46.407 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.405+0000 7fd913a67700 1 -- 172.21.15.174:0/1218805456 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90c0f5990 con 0x7fd90c0f47c0 2023-12-25T21:10:46.407 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.405+0000 7fd9027fc700 1 -- 172.21.15.174:0/1218805456 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd8fc007e70 con 0x7fd90c0f47c0 2023-12-25T21:10:46.407 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.405+0000 7fd9027fc700 1 -- 172.21.15.174:0/1218805456 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8fc014ea0 con 0x7fd90c0f47c0 2023-12-25T21:10:46.407 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.405+0000 7fd913a67700 1 -- 172.21.15.174:0/1218805456 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90c0f5e20 con 0x7fd90c0f47c0 2023-12-25T21:10:46.408 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.406+0000 7fd9027fc700 1 -- 172.21.15.174:0/1218805456 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd8fc01d4e0 con 0x7fd90c0f47c0 2023-12-25T21:10:46.408 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.406+0000 7fd9027fc700 1 --2- 172.21.15.174:0/1218805456 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd8f802ff60 0x7fd8f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:46.408 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.406+0000 7fd9027fc700 1 -- 172.21.15.174:0/1218805456 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd8fc040b80 con 0x7fd90c0f47c0 2023-12-25T21:10:46.408 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.406+0000 7fd913a67700 1 -- 172.21.15.174:0/1218805456 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90c059100 con 0x7fd90c0f47c0 2023-12-25T21:10:46.408 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.406+0000 7fd911002700 1 --2- 172.21.15.174:0/1218805456 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd8f802ff60 0x7fd8f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:46.408 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.407+0000 7fd911002700 1 --2- 172.21.15.174:0/1218805456 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd8f802ff60 0x7fd8f8032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fd9080096a0 tx=0x7fd908006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:46.411 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.411+0000 7fd9027fc700 1 -- 172.21.15.174:0/1218805456 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd8fc025e60 con 0x7fd90c0f47c0 2023-12-25T21:10:46.619 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.619+0000 7fd913a67700 1 -- 172.21.15.174:0/1218805456 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd90c045ad0 con 0x7fd90c0f47c0 2023-12-25T21:10:46.620 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.619+0000 7fd9027fc700 1 -- 172.21.15.174:0/1218805456 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd8fc025aa0 con 0x7fd90c0f47c0 2023-12-25T21:10:46.620 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:46.620 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:46.622 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.621+0000 7fd913a67700 1 -- 172.21.15.174:0/1218805456 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd8f802ff60 msgr2=0x7fd8f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:46.622 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.621+0000 7fd913a67700 1 --2- 172.21.15.174:0/1218805456 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd8f802ff60 0x7fd8f8032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fd9080096a0 tx=0x7fd908006b40).stop 2023-12-25T21:10:46.622 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.621+0000 7fd913a67700 1 -- 172.21.15.174:0/1218805456 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90c0f47c0 msgr2=0x7fd90c0f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:46.622 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.621+0000 7fd913a67700 1 --2- 172.21.15.174:0/1218805456 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90c0f47c0 0x7fd90c0f7370 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fd8fc014000 tx=0x7fd8fc00b040).stop 2023-12-25T21:10:46.622 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.621+0000 7fd913a67700 1 -- 172.21.15.174:0/1218805456 shutdown_connections 2023-12-25T21:10:46.622 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.621+0000 7fd913a67700 1 --2- 172.21.15.174:0/1218805456 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd8f802ff60 0x7fd8f8032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:46.622 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.621+0000 7fd913a67700 1 --2- 172.21.15.174:0/1218805456 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90c0f47c0 0x7fd90c0f7370 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:46.622 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.621+0000 7fd913a67700 1 -- 172.21.15.174:0/1218805456 >> 172.21.15.174:0/1218805456 conn(0x7fd90c0f02e0 msgr2=0x7fd90c0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:46.622 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.621+0000 7fd913a67700 1 -- 172.21.15.174:0/1218805456 shutdown_connections 2023-12-25T21:10:46.623 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:46.621+0000 7fd913a67700 1 -- 172.21.15.174:0/1218805456 wait complete. 2023-12-25T21:10:46.623 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:46.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:46 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:44.602923+0000 mgr.smithi061.jxxeke (mgr.14164) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:47.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:47 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:46.620860+0000 mon.smithi061 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.174:0/1218805456' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:48.032 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:48.032 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:48.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:48 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:46.603284+0000 mgr.smithi061.jxxeke (mgr.14164) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:49.177 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.173+0000 7f6415d02700 1 -- 172.21.15.174:0/1787426046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64100f6510 msgr2=0x7f64100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:49.178 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.173+0000 7f6415d02700 1 --2- 172.21.15.174:0/1787426046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64100f6510 0x7f64100f6930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f63f8009fb0 tx=0x7f63f8009b30).stop 2023-12-25T21:10:49.178 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.173+0000 7f6415d02700 1 -- 172.21.15.174:0/1787426046 shutdown_connections 2023-12-25T21:10:49.178 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.173+0000 7f6415d02700 1 --2- 172.21.15.174:0/1787426046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64100f6510 0x7f64100f6930 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:49.178 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.173+0000 7f6415d02700 1 -- 172.21.15.174:0/1787426046 >> 172.21.15.174:0/1787426046 conn(0x7f64100f1c70 msgr2=0x7f64100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:49.178 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.173+0000 7f6415d02700 1 -- 172.21.15.174:0/1787426046 shutdown_connections 2023-12-25T21:10:49.179 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.173+0000 7f6415d02700 1 -- 172.21.15.174:0/1787426046 wait complete. 2023-12-25T21:10:49.179 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.174+0000 7f6415d02700 1 Processor -- start 2023-12-25T21:10:49.179 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.174+0000 7f6415d02700 1 -- start start 2023-12-25T21:10:49.179 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.174+0000 7f6415d02700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64100f6510 0x7f64101066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:49.179 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.174+0000 7f6415d02700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6410106bc0 con 0x7f64100f6510 2023-12-25T21:10:49.179 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.175+0000 7f640f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64100f6510 0x7f64101066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:49.180 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.175+0000 7f640f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64100f6510 0x7f64101066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:36142/0 (socket says 172.21.15.174:36142) 2023-12-25T21:10:49.180 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.175+0000 7f640f7fe700 1 -- 172.21.15.174:0/3885119563 learned_addr learned my addr 172.21.15.174:0/3885119563 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:49.180 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.175+0000 7f640f7fe700 1 -- 172.21.15.174:0/3885119563 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f8005040 con 0x7f64100f6510 2023-12-25T21:10:49.180 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.175+0000 7f640f7fe700 1 --2- 172.21.15.174:0/3885119563 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64100f6510 0x7f64101066d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f63f8013510 tx=0x7f63f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:49.180 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.175+0000 7f6407fff700 1 -- 172.21.15.174:0/3885119563 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63f8009320 con 0x7f64100f6510 2023-12-25T21:10:49.180 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.175+0000 7f6415d02700 1 -- 172.21.15.174:0/3885119563 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6410106dc0 con 0x7f64100f6510 2023-12-25T21:10:49.180 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.175+0000 7f6407fff700 1 -- 172.21.15.174:0/3885119563 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63f8013f00 con 0x7f64100f6510 2023-12-25T21:10:49.180 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.175+0000 7f6407fff700 1 -- 172.21.15.174:0/3885119563 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63f8014d90 con 0x7f64100f6510 2023-12-25T21:10:49.181 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.175+0000 7f6415d02700 1 -- 172.21.15.174:0/3885119563 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6410107250 con 0x7f64100f6510 2023-12-25T21:10:49.181 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.176+0000 7f6407fff700 1 -- 172.21.15.174:0/3885119563 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63f80095f0 con 0x7f64100f6510 2023-12-25T21:10:49.181 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.176+0000 7f6415d02700 1 -- 172.21.15.174:0/3885119563 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6410045ad0 con 0x7f64100f6510 2023-12-25T21:10:49.181 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.176+0000 7f6407fff700 1 --2- 172.21.15.174:0/3885119563 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f63fc02ff60 0x7f63fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:49.181 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.176+0000 7f6407fff700 1 -- 172.21.15.174:0/3885119563 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f63f8041000 con 0x7f64100f6510 2023-12-25T21:10:49.181 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.176+0000 7f640effd700 1 --2- 172.21.15.174:0/3885119563 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f63fc02ff60 0x7f63fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:49.181 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.177+0000 7f640effd700 1 --2- 172.21.15.174:0/3885119563 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f63fc02ff60 0x7f63fc032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f64000096a0 tx=0x7f6400006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:49.182 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.181+0000 7f6407fff700 1 -- 172.21.15.174:0/3885119563 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63f8014420 con 0x7f64100f6510 2023-12-25T21:10:49.388 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.387+0000 7f6415d02700 1 -- 172.21.15.174:0/3885119563 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6410108100 con 0x7f64100f6510 2023-12-25T21:10:49.389 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.388+0000 7f6407fff700 1 -- 172.21.15.174:0/3885119563 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f63f8010230 con 0x7f64100f6510 2023-12-25T21:10:49.389 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:49.389 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:49.390 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.390+0000 7f6415d02700 1 -- 172.21.15.174:0/3885119563 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f63fc02ff60 msgr2=0x7f63fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:49.391 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.390+0000 7f6415d02700 1 --2- 172.21.15.174:0/3885119563 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f63fc02ff60 0x7f63fc032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f64000096a0 tx=0x7f6400006b40).stop 2023-12-25T21:10:49.391 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.390+0000 7f6415d02700 1 -- 172.21.15.174:0/3885119563 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64100f6510 msgr2=0x7f64101066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:49.391 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.390+0000 7f6415d02700 1 --2- 172.21.15.174:0/3885119563 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64100f6510 0x7f64101066d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f63f8013510 tx=0x7f63f800b040).stop 2023-12-25T21:10:49.391 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.390+0000 7f6415d02700 1 -- 172.21.15.174:0/3885119563 shutdown_connections 2023-12-25T21:10:49.391 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.390+0000 7f6415d02700 1 --2- 172.21.15.174:0/3885119563 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f63fc02ff60 0x7f63fc032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:49.391 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.390+0000 7f6415d02700 1 --2- 172.21.15.174:0/3885119563 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64100f6510 0x7f64101066d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:49.391 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.390+0000 7f6415d02700 1 -- 172.21.15.174:0/3885119563 >> 172.21.15.174:0/3885119563 conn(0x7f64100f1c70 msgr2=0x7f64100f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:49.391 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.390+0000 7f6415d02700 1 -- 172.21.15.174:0/3885119563 shutdown_connections 2023-12-25T21:10:49.391 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:49.390+0000 7f6415d02700 1 -- 172.21.15.174:0/3885119563 wait complete. 2023-12-25T21:10:49.391 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:50.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:50 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:48.603622+0000 mgr.smithi061.jxxeke (mgr.14164) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:50.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:50 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:49.389452+0000 mon.smithi061 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.174:0/3885119563' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:50.761 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:50.761 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:51.963 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.962+0000 7ff757502700 1 -- 172.21.15.174:0/1726047568 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7500f5410 msgr2=0x7ff7500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:51.964 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.962+0000 7ff757502700 1 --2- 172.21.15.174:0/1726047568 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7500f5410 0x7ff7500f5830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7ff740009fb0 tx=0x7ff740009b30).stop 2023-12-25T21:10:51.964 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.962+0000 7ff757502700 1 -- 172.21.15.174:0/1726047568 shutdown_connections 2023-12-25T21:10:51.964 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.962+0000 7ff757502700 1 --2- 172.21.15.174:0/1726047568 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7500f5410 0x7ff7500f5830 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:51.964 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.962+0000 7ff757502700 1 -- 172.21.15.174:0/1726047568 >> 172.21.15.174:0/1726047568 conn(0x7ff7500f0b70 msgr2=0x7ff7500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:51.964 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.962+0000 7ff757502700 1 -- 172.21.15.174:0/1726047568 shutdown_connections 2023-12-25T21:10:51.964 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.962+0000 7ff757502700 1 -- 172.21.15.174:0/1726047568 wait complete. 2023-12-25T21:10:51.964 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.963+0000 7ff757502700 1 Processor -- start 2023-12-25T21:10:51.964 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.963+0000 7ff757502700 1 -- start start 2023-12-25T21:10:51.964 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.963+0000 7ff757502700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7500f5410 0x7ff750105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:51.964 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.963+0000 7ff757502700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff750105b20 con 0x7ff7500f5410 2023-12-25T21:10:51.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.963+0000 7ff75529e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7500f5410 0x7ff750105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:51.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.963+0000 7ff75529e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7500f5410 0x7ff750105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:36160/0 (socket says 172.21.15.174:36160) 2023-12-25T21:10:51.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.963+0000 7ff75529e700 1 -- 172.21.15.174:0/400592253 learned_addr learned my addr 172.21.15.174:0/400592253 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:51.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.964+0000 7ff75529e700 1 -- 172.21.15.174:0/400592253 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff740005040 con 0x7ff7500f5410 2023-12-25T21:10:51.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.964+0000 7ff75529e700 1 --2- 172.21.15.174:0/400592253 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7500f5410 0x7ff750105630 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7ff740013510 tx=0x7ff74000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:51.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.964+0000 7ff745ffb700 1 -- 172.21.15.174:0/400592253 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff740009320 con 0x7ff7500f5410 2023-12-25T21:10:51.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.964+0000 7ff757502700 1 -- 172.21.15.174:0/400592253 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff750105d20 con 0x7ff7500f5410 2023-12-25T21:10:51.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.964+0000 7ff745ffb700 1 -- 172.21.15.174:0/400592253 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff740013f00 con 0x7ff7500f5410 2023-12-25T21:10:51.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.964+0000 7ff745ffb700 1 -- 172.21.15.174:0/400592253 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff740014d90 con 0x7ff7500f5410 2023-12-25T21:10:51.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.964+0000 7ff757502700 1 -- 172.21.15.174:0/400592253 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7501061b0 con 0x7ff7500f5410 2023-12-25T21:10:51.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.965+0000 7ff745ffb700 1 -- 172.21.15.174:0/400592253 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7400263a0 con 0x7ff7500f5410 2023-12-25T21:10:51.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.965+0000 7ff757502700 1 -- 172.21.15.174:0/400592253 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff750045ad0 con 0x7ff7500f5410 2023-12-25T21:10:51.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.965+0000 7ff745ffb700 1 --2- 172.21.15.174:0/400592253 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff73c02ffb0 0x7ff73c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:51.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.965+0000 7ff745ffb700 1 -- 172.21.15.174:0/400592253 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff740040fc0 con 0x7ff7500f5410 2023-12-25T21:10:51.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.965+0000 7ff754a9d700 1 --2- 172.21.15.174:0/400592253 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff73c02ffb0 0x7ff73c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:51.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.966+0000 7ff754a9d700 1 --2- 172.21.15.174:0/400592253 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff73c02ffb0 0x7ff73c032470 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7ff74c0096a0 tx=0x7ff74c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:51.971 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:51.970+0000 7ff745ffb700 1 -- 172.21.15.174:0/400592253 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff74006d2b0 con 0x7ff7500f5410 2023-12-25T21:10:52.175 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:52.174+0000 7ff757502700 1 -- 172.21.15.174:0/400592253 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7500fae60 con 0x7ff7500f5410 2023-12-25T21:10:52.175 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:52.174+0000 7ff745ffb700 1 -- 172.21.15.174:0/400592253 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff7400126f0 con 0x7ff7500f5410 2023-12-25T21:10:52.175 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:52.175 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:52.177 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:52.176+0000 7ff757502700 1 -- 172.21.15.174:0/400592253 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff73c02ffb0 msgr2=0x7ff73c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:52.177 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:52.176+0000 7ff757502700 1 --2- 172.21.15.174:0/400592253 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff73c02ffb0 0x7ff73c032470 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7ff74c0096a0 tx=0x7ff74c006b40).stop 2023-12-25T21:10:52.177 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:52.176+0000 7ff757502700 1 -- 172.21.15.174:0/400592253 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7500f5410 msgr2=0x7ff750105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:52.177 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:52.176+0000 7ff757502700 1 --2- 172.21.15.174:0/400592253 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7500f5410 0x7ff750105630 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7ff740013510 tx=0x7ff74000b040).stop 2023-12-25T21:10:52.177 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:52.176+0000 7ff757502700 1 -- 172.21.15.174:0/400592253 shutdown_connections 2023-12-25T21:10:52.177 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:52.176+0000 7ff757502700 1 --2- 172.21.15.174:0/400592253 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff73c02ffb0 0x7ff73c032470 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:52.177 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:52.176+0000 7ff757502700 1 --2- 172.21.15.174:0/400592253 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7500f5410 0x7ff750105630 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:52.178 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:52.176+0000 7ff757502700 1 -- 172.21.15.174:0/400592253 >> 172.21.15.174:0/400592253 conn(0x7ff7500f0b70 msgr2=0x7ff7500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:52.178 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:52.176+0000 7ff757502700 1 -- 172.21.15.174:0/400592253 shutdown_connections 2023-12-25T21:10:52.178 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:52.176+0000 7ff757502700 1 -- 172.21.15.174:0/400592253 wait complete. 2023-12-25T21:10:52.178 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:52.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:52 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:50.604009+0000 mgr.smithi061.jxxeke (mgr.14164) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:52.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:52 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:52.175984+0000 mon.smithi061 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.174:0/400592253' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:53.572 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:53.572 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:53.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:52.705004+0000 mon.smithi061 (mon.0) 200 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/mirror_snapshot_schedule"}]: dispatch 2023-12-25T21:10:53.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:52.706328+0000 mon.smithi061 (mon.0) 201 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/trash_purge_schedule"}]: dispatch 2023-12-25T21:10:54.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:54 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:52.604342+0000 mgr.smithi061.jxxeke (mgr.14164) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:54.770 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.768+0000 7ffaf0c0a700 1 -- 172.21.15.174:0/2407572373 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffaec0f5410 msgr2=0x7ffaec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:54.770 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.768+0000 7ffaf0c0a700 1 --2- 172.21.15.174:0/2407572373 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffaec0f5410 0x7ffaec0f5830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7ffad4009fb0 tx=0x7ffad4009b30).stop 2023-12-25T21:10:54.770 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.768+0000 7ffaf0c0a700 1 -- 172.21.15.174:0/2407572373 shutdown_connections 2023-12-25T21:10:54.770 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.768+0000 7ffaf0c0a700 1 --2- 172.21.15.174:0/2407572373 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffaec0f5410 0x7ffaec0f5830 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:54.770 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.768+0000 7ffaf0c0a700 1 -- 172.21.15.174:0/2407572373 >> 172.21.15.174:0/2407572373 conn(0x7ffaec0f0b70 msgr2=0x7ffaec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:54.771 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.768+0000 7ffaf0c0a700 1 -- 172.21.15.174:0/2407572373 shutdown_connections 2023-12-25T21:10:54.771 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.768+0000 7ffaf0c0a700 1 -- 172.21.15.174:0/2407572373 wait complete. 2023-12-25T21:10:54.771 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.768+0000 7ffaf0c0a700 1 Processor -- start 2023-12-25T21:10:54.771 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.768+0000 7ffaf0c0a700 1 -- start start 2023-12-25T21:10:54.771 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.768+0000 7ffaf0c0a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffaec0f5410 0x7ffaec10b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:54.771 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.768+0000 7ffaf0c0a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec107240 con 0x7ffaec0f5410 2023-12-25T21:10:54.771 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.769+0000 7ffaea59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffaec0f5410 0x7ffaec10b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:54.771 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.769+0000 7ffaea59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffaec0f5410 0x7ffaec10b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:36178/0 (socket says 172.21.15.174:36178) 2023-12-25T21:10:54.771 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.769+0000 7ffaea59c700 1 -- 172.21.15.174:0/112683831 learned_addr learned my addr 172.21.15.174:0/112683831 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:54.771 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.769+0000 7ffaea59c700 1 -- 172.21.15.174:0/112683831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad4005040 con 0x7ffaec0f5410 2023-12-25T21:10:54.771 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.769+0000 7ffaea59c700 1 --2- 172.21.15.174:0/112683831 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffaec0f5410 0x7ffaec10b850 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7ffad4013510 tx=0x7ffad400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:54.771 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.769+0000 7ffae2ffd700 1 -- 172.21.15.174:0/112683831 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffad4009320 con 0x7ffaec0f5410 2023-12-25T21:10:54.771 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.769+0000 7ffaf0c0a700 1 -- 172.21.15.174:0/112683831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaec060220 con 0x7ffaec0f5410 2023-12-25T21:10:54.771 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.769+0000 7ffae2ffd700 1 -- 172.21.15.174:0/112683831 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffad4013f00 con 0x7ffaec0f5410 2023-12-25T21:10:54.772 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.769+0000 7ffae2ffd700 1 -- 172.21.15.174:0/112683831 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffad4014d90 con 0x7ffaec0f5410 2023-12-25T21:10:54.772 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.769+0000 7ffaf0c0a700 1 -- 172.21.15.174:0/112683831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaec10bd40 con 0x7ffaec0f5410 2023-12-25T21:10:54.772 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.770+0000 7ffae2ffd700 1 -- 172.21.15.174:0/112683831 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffad40095f0 con 0x7ffaec0f5410 2023-12-25T21:10:54.773 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.770+0000 7ffaf0c0a700 1 -- 172.21.15.174:0/112683831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaec045ad0 con 0x7ffaec0f5410 2023-12-25T21:10:54.773 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.770+0000 7ffae2ffd700 1 --2- 172.21.15.174:0/112683831 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ffad802ff60 0x7ffad8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:54.773 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.770+0000 7ffae2ffd700 1 -- 172.21.15.174:0/112683831 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffad40410a0 con 0x7ffaec0f5410 2023-12-25T21:10:54.773 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.770+0000 7ffae9d9b700 1 --2- 172.21.15.174:0/112683831 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ffad802ff60 0x7ffad8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:54.773 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.771+0000 7ffae9d9b700 1 --2- 172.21.15.174:0/112683831 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ffad802ff60 0x7ffad8032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7ffadc0096a0 tx=0x7ffadc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:54.775 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.775+0000 7ffae2ffd700 1 -- 172.21.15.174:0/112683831 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffad401d7f0 con 0x7ffaec0f5410 2023-12-25T21:10:54.980 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.979+0000 7ffaf0c0a700 1 -- 172.21.15.174:0/112683831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffaec0fae60 con 0x7ffaec0f5410 2023-12-25T21:10:54.981 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.980+0000 7ffae2ffd700 1 -- 172.21.15.174:0/112683831 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffad4010230 con 0x7ffaec0f5410 2023-12-25T21:10:54.981 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:54.981 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:54.982 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.982+0000 7ffaf0c0a700 1 -- 172.21.15.174:0/112683831 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ffad802ff60 msgr2=0x7ffad8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:54.983 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.982+0000 7ffaf0c0a700 1 --2- 172.21.15.174:0/112683831 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ffad802ff60 0x7ffad8032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7ffadc0096a0 tx=0x7ffadc006b40).stop 2023-12-25T21:10:54.983 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.982+0000 7ffaf0c0a700 1 -- 172.21.15.174:0/112683831 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffaec0f5410 msgr2=0x7ffaec10b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:54.983 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.982+0000 7ffaf0c0a700 1 --2- 172.21.15.174:0/112683831 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffaec0f5410 0x7ffaec10b850 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7ffad4013510 tx=0x7ffad400b040).stop 2023-12-25T21:10:54.983 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.982+0000 7ffaf0c0a700 1 -- 172.21.15.174:0/112683831 shutdown_connections 2023-12-25T21:10:54.983 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.982+0000 7ffaf0c0a700 1 --2- 172.21.15.174:0/112683831 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ffad802ff60 0x7ffad8032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:54.983 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.982+0000 7ffaf0c0a700 1 --2- 172.21.15.174:0/112683831 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffaec0f5410 0x7ffaec10b850 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:54.983 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.982+0000 7ffaf0c0a700 1 -- 172.21.15.174:0/112683831 >> 172.21.15.174:0/112683831 conn(0x7ffaec0f0b70 msgr2=0x7ffaec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:54.983 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.982+0000 7ffaf0c0a700 1 -- 172.21.15.174:0/112683831 shutdown_connections 2023-12-25T21:10:54.983 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:54.982+0000 7ffaf0c0a700 1 -- 172.21.15.174:0/112683831 wait complete. 2023-12-25T21:10:54.983 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:55.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:55 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:54.981557+0000 mon.smithi061 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.174:0/112683831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:56.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:56.367 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:10:56.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:56 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:54.604724+0000 mgr.smithi061.jxxeke (mgr.14164) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:57.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.536+0000 7f76332e7700 1 -- 172.21.15.174:0/581796027 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f762c0f5410 msgr2=0x7f762c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:57.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.536+0000 7f76332e7700 1 --2- 172.21.15.174:0/581796027 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f762c0f5410 0x7f762c0f5830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f761c009fb0 tx=0x7f761c009b30).stop 2023-12-25T21:10:57.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.536+0000 7f76332e7700 1 -- 172.21.15.174:0/581796027 shutdown_connections 2023-12-25T21:10:57.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.536+0000 7f76332e7700 1 --2- 172.21.15.174:0/581796027 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f762c0f5410 0x7f762c0f5830 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:57.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.536+0000 7f76332e7700 1 -- 172.21.15.174:0/581796027 >> 172.21.15.174:0/581796027 conn(0x7f762c0f0b70 msgr2=0x7f762c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:57.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.536+0000 7f76332e7700 1 -- 172.21.15.174:0/581796027 shutdown_connections 2023-12-25T21:10:57.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.536+0000 7f76332e7700 1 -- 172.21.15.174:0/581796027 wait complete. 2023-12-25T21:10:57.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.537+0000 7f76332e7700 1 Processor -- start 2023-12-25T21:10:57.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.537+0000 7f76332e7700 1 -- start start 2023-12-25T21:10:57.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.537+0000 7f76332e7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f762c0f5410 0x7f762c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:57.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.537+0000 7f76332e7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c0fb820 con 0x7f762c0f5410 2023-12-25T21:10:57.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.537+0000 7f7631083700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f762c0f5410 0x7f762c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:57.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.537+0000 7f7631083700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f762c0f5410 0x7f762c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:51326/0 (socket says 172.21.15.174:51326) 2023-12-25T21:10:57.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.537+0000 7f7631083700 1 -- 172.21.15.174:0/2456480770 learned_addr learned my addr 172.21.15.174:0/2456480770 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:10:57.540 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.537+0000 7f7631083700 1 -- 172.21.15.174:0/2456480770 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c005040 con 0x7f762c0f5410 2023-12-25T21:10:57.540 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.537+0000 7f7631083700 1 --2- 172.21.15.174:0/2456480770 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f762c0f5410 0x7f762c0fd3a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f761c013510 tx=0x7f761c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:57.540 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.538+0000 7f7621ffb700 1 -- 172.21.15.174:0/2456480770 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f761c009320 con 0x7f762c0f5410 2023-12-25T21:10:57.540 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.538+0000 7f76332e7700 1 -- 172.21.15.174:0/2456480770 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762c0fb9c0 con 0x7f762c0f5410 2023-12-25T21:10:57.540 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.538+0000 7f7621ffb700 1 -- 172.21.15.174:0/2456480770 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f761c013f00 con 0x7f762c0f5410 2023-12-25T21:10:57.541 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.538+0000 7f7621ffb700 1 -- 172.21.15.174:0/2456480770 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f761c014d90 con 0x7f762c0f5410 2023-12-25T21:10:57.541 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.538+0000 7f76332e7700 1 -- 172.21.15.174:0/2456480770 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762c0fbe50 con 0x7f762c0f5410 2023-12-25T21:10:57.541 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.539+0000 7f7621ffb700 1 -- 172.21.15.174:0/2456480770 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f761c0095f0 con 0x7f762c0f5410 2023-12-25T21:10:57.541 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.539+0000 7f76332e7700 1 -- 172.21.15.174:0/2456480770 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f762c059100 con 0x7f762c0f5410 2023-12-25T21:10:57.541 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.539+0000 7f7621ffb700 1 --2- 172.21.15.174:0/2456480770 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f761802ff60 0x7f7618032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:10:57.541 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.539+0000 7f7621ffb700 1 -- 172.21.15.174:0/2456480770 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f761c041080 con 0x7f762c0f5410 2023-12-25T21:10:57.541 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.539+0000 7f7630882700 1 --2- 172.21.15.174:0/2456480770 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f761802ff60 0x7f7618032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:10:57.541 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.539+0000 7f7630882700 1 --2- 172.21.15.174:0/2456480770 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f761802ff60 0x7f7618032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f76280096a0 tx=0x7f7628006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:10:57.544 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.544+0000 7f7621ffb700 1 -- 172.21.15.174:0/2456480770 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f761c014420 con 0x7f762c0f5410 2023-12-25T21:10:57.752 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.751+0000 7f76332e7700 1 -- 172.21.15.174:0/2456480770 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f762c045ad0 con 0x7f762c0f5410 2023-12-25T21:10:57.752 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.751+0000 7f7621ffb700 1 -- 172.21.15.174:0/2456480770 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f761c010230 con 0x7f762c0f5410 2023-12-25T21:10:57.752 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:10:57.752 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:10:57.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.753+0000 7f76332e7700 1 -- 172.21.15.174:0/2456480770 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f761802ff60 msgr2=0x7f7618032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:57.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.753+0000 7f76332e7700 1 --2- 172.21.15.174:0/2456480770 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f761802ff60 0x7f7618032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f76280096a0 tx=0x7f7628006b40).stop 2023-12-25T21:10:57.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.753+0000 7f76332e7700 1 -- 172.21.15.174:0/2456480770 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f762c0f5410 msgr2=0x7f762c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:10:57.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.753+0000 7f76332e7700 1 --2- 172.21.15.174:0/2456480770 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f762c0f5410 0x7f762c0fd3a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f761c013510 tx=0x7f761c00b040).stop 2023-12-25T21:10:57.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.753+0000 7f76332e7700 1 -- 172.21.15.174:0/2456480770 shutdown_connections 2023-12-25T21:10:57.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.753+0000 7f76332e7700 1 --2- 172.21.15.174:0/2456480770 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f761802ff60 0x7f7618032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:57.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.753+0000 7f76332e7700 1 --2- 172.21.15.174:0/2456480770 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f762c0f5410 0x7f762c0fd3a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:10:57.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.753+0000 7f76332e7700 1 -- 172.21.15.174:0/2456480770 >> 172.21.15.174:0/2456480770 conn(0x7f762c0f0b70 msgr2=0x7f762c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:10:57.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.753+0000 7f76332e7700 1 -- 172.21.15.174:0/2456480770 shutdown_connections 2023-12-25T21:10:57.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:10:57.753+0000 7f76332e7700 1 -- 172.21.15.174:0/2456480770 wait complete. 2023-12-25T21:10:57.755 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:10:58.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:58 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:56.605044+0000 mgr.smithi061.jxxeke (mgr.14164) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:10:58.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:10:58 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:10:57.752741+0000 mon.smithi061 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.174:0/2456480770' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:10:59.154 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:10:59.154 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:00.323 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.321+0000 7f8368ecd700 1 -- 172.21.15.174:0/3869752195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83640f5410 msgr2=0x7f83640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:00.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.321+0000 7f8368ecd700 1 --2- 172.21.15.174:0/3869752195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83640f5410 0x7f83640f5830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f834c009fb0 tx=0x7f834c009b30).stop 2023-12-25T21:11:00.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.321+0000 7f8368ecd700 1 -- 172.21.15.174:0/3869752195 shutdown_connections 2023-12-25T21:11:00.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.321+0000 7f8368ecd700 1 --2- 172.21.15.174:0/3869752195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83640f5410 0x7f83640f5830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:00.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.321+0000 7f8368ecd700 1 -- 172.21.15.174:0/3869752195 >> 172.21.15.174:0/3869752195 conn(0x7f83640f0b70 msgr2=0x7f83640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:00.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.321+0000 7f8368ecd700 1 -- 172.21.15.174:0/3869752195 shutdown_connections 2023-12-25T21:11:00.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.321+0000 7f8368ecd700 1 -- 172.21.15.174:0/3869752195 wait complete. 2023-12-25T21:11:00.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.322+0000 7f8368ecd700 1 Processor -- start 2023-12-25T21:11:00.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.322+0000 7f8368ecd700 1 -- start start 2023-12-25T21:11:00.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.322+0000 7f8368ecd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83640f5410 0x7f8364105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:00.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.322+0000 7f8368ecd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8364105b20 con 0x7f83640f5410 2023-12-25T21:11:00.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.322+0000 7f836259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83640f5410 0x7f8364105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:00.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.322+0000 7f836259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83640f5410 0x7f8364105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:51346/0 (socket says 172.21.15.174:51346) 2023-12-25T21:11:00.325 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.322+0000 7f836259c700 1 -- 172.21.15.174:0/60616807 learned_addr learned my addr 172.21.15.174:0/60616807 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:00.325 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.322+0000 7f836259c700 1 -- 172.21.15.174:0/60616807 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834c005040 con 0x7f83640f5410 2023-12-25T21:11:00.325 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.322+0000 7f836259c700 1 --2- 172.21.15.174:0/60616807 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83640f5410 0x7f8364105630 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f834c013510 tx=0x7f834c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:00.325 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.323+0000 7f835affd700 1 -- 172.21.15.174:0/60616807 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f834c009320 con 0x7f83640f5410 2023-12-25T21:11:00.325 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.323+0000 7f8368ecd700 1 -- 172.21.15.174:0/60616807 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8364105d20 con 0x7f83640f5410 2023-12-25T21:11:00.325 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.323+0000 7f835affd700 1 -- 172.21.15.174:0/60616807 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f834c013f00 con 0x7f83640f5410 2023-12-25T21:11:00.325 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.323+0000 7f835affd700 1 -- 172.21.15.174:0/60616807 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f834c014d90 con 0x7f83640f5410 2023-12-25T21:11:00.325 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.323+0000 7f8368ecd700 1 -- 172.21.15.174:0/60616807 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83641061b0 con 0x7f83640f5410 2023-12-25T21:11:00.326 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.323+0000 7f835affd700 1 -- 172.21.15.174:0/60616807 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f834c0095f0 con 0x7f83640f5410 2023-12-25T21:11:00.326 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.324+0000 7f8368ecd700 1 -- 172.21.15.174:0/60616807 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8364045ad0 con 0x7f83640f5410 2023-12-25T21:11:00.326 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.324+0000 7f835affd700 1 --2- 172.21.15.174:0/60616807 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f835002ff60 0x7f8350032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:00.326 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.324+0000 7f835affd700 1 -- 172.21.15.174:0/60616807 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f834c042e90 con 0x7f83640f5410 2023-12-25T21:11:00.326 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.324+0000 7f8361d9b700 1 --2- 172.21.15.174:0/60616807 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f835002ff60 0x7f8350032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:00.326 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.324+0000 7f8361d9b700 1 --2- 172.21.15.174:0/60616807 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f835002ff60 0x7f8350032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f83540096a0 tx=0x7f8354006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:00.329 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.328+0000 7f835affd700 1 -- 172.21.15.174:0/60616807 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f834c014420 con 0x7f83640f5410 2023-12-25T21:11:00.535 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.535+0000 7f8368ecd700 1 -- 172.21.15.174:0/60616807 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83640fae60 con 0x7f83640f5410 2023-12-25T21:11:00.536 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.535+0000 7f835affd700 1 -- 172.21.15.174:0/60616807 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f834c011290 con 0x7f83640f5410 2023-12-25T21:11:00.536 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:00.536 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:00.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.537+0000 7f8368ecd700 1 -- 172.21.15.174:0/60616807 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f835002ff60 msgr2=0x7f8350032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:00.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.537+0000 7f8368ecd700 1 --2- 172.21.15.174:0/60616807 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f835002ff60 0x7f8350032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f83540096a0 tx=0x7f8354006b40).stop 2023-12-25T21:11:00.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.537+0000 7f8368ecd700 1 -- 172.21.15.174:0/60616807 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83640f5410 msgr2=0x7f8364105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:00.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.537+0000 7f8368ecd700 1 --2- 172.21.15.174:0/60616807 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83640f5410 0x7f8364105630 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f834c013510 tx=0x7f834c00b040).stop 2023-12-25T21:11:00.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.537+0000 7f8368ecd700 1 -- 172.21.15.174:0/60616807 shutdown_connections 2023-12-25T21:11:00.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.537+0000 7f8368ecd700 1 --2- 172.21.15.174:0/60616807 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f835002ff60 0x7f8350032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:00.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.537+0000 7f8368ecd700 1 --2- 172.21.15.174:0/60616807 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83640f5410 0x7f8364105630 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:00.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.537+0000 7f8368ecd700 1 -- 172.21.15.174:0/60616807 >> 172.21.15.174:0/60616807 conn(0x7f83640f0b70 msgr2=0x7f83640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:00.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.537+0000 7f8368ecd700 1 -- 172.21.15.174:0/60616807 shutdown_connections 2023-12-25T21:11:00.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:00.537+0000 7f8368ecd700 1 -- 172.21.15.174:0/60616807 wait complete. 2023-12-25T21:11:00.539 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:00.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:00 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:10:58.605366+0000 mgr.smithi061.jxxeke (mgr.14164) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:01.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:01 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:00.536767+0000 mon.smithi061 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.174:0/60616807' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:01.942 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:01.943 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:02.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:02 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:00.605646+0000 mgr.smithi061.jxxeke (mgr.14164) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:03.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.146+0000 7f55605d6700 1 -- 172.21.15.174:0/489806552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55580f4740 msgr2=0x7f55580f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:03.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.146+0000 7f55605d6700 1 --2- 172.21.15.174:0/489806552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55580f4740 0x7f55580f4b60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f5554009fb0 tx=0x7f5554009b30).stop 2023-12-25T21:11:03.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.146+0000 7f55605d6700 1 -- 172.21.15.174:0/489806552 shutdown_connections 2023-12-25T21:11:03.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.146+0000 7f55605d6700 1 --2- 172.21.15.174:0/489806552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55580f4740 0x7f55580f4b60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:03.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.146+0000 7f55605d6700 1 -- 172.21.15.174:0/489806552 >> 172.21.15.174:0/489806552 conn(0x7f55580f0260 msgr2=0x7f55580f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:03.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.146+0000 7f55605d6700 1 -- 172.21.15.174:0/489806552 shutdown_connections 2023-12-25T21:11:03.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.147+0000 7f55605d6700 1 -- 172.21.15.174:0/489806552 wait complete. 2023-12-25T21:11:03.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.147+0000 7f55605d6700 1 Processor -- start 2023-12-25T21:11:03.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.147+0000 7f55605d6700 1 -- start start 2023-12-25T21:11:03.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.147+0000 7f55605d6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55580f4740 0x7f55581066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:03.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.147+0000 7f55605d6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5558106ba0 con 0x7f55580f4740 2023-12-25T21:11:03.150 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.147+0000 7f555e372700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55580f4740 0x7f55581066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:03.150 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.147+0000 7f555e372700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55580f4740 0x7f55581066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:51368/0 (socket says 172.21.15.174:51368) 2023-12-25T21:11:03.150 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.147+0000 7f555e372700 1 -- 172.21.15.174:0/2106993017 learned_addr learned my addr 172.21.15.174:0/2106993017 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:03.150 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.148+0000 7f555e372700 1 -- 172.21.15.174:0/2106993017 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5554005040 con 0x7f55580f4740 2023-12-25T21:11:03.150 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.148+0000 7f555e372700 1 --2- 172.21.15.174:0/2106993017 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55580f4740 0x7f55581066b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f5554013510 tx=0x7f555400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:03.150 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.148+0000 7f554effd700 1 -- 172.21.15.174:0/2106993017 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5554009320 con 0x7f55580f4740 2023-12-25T21:11:03.150 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.148+0000 7f55605d6700 1 -- 172.21.15.174:0/2106993017 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5558106da0 con 0x7f55580f4740 2023-12-25T21:11:03.150 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.148+0000 7f554effd700 1 -- 172.21.15.174:0/2106993017 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5554013f00 con 0x7f55580f4740 2023-12-25T21:11:03.150 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.148+0000 7f554effd700 1 -- 172.21.15.174:0/2106993017 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5554014d90 con 0x7f55580f4740 2023-12-25T21:11:03.150 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.148+0000 7f55605d6700 1 -- 172.21.15.174:0/2106993017 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5558107230 con 0x7f55580f4740 2023-12-25T21:11:03.151 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.149+0000 7f554effd700 1 -- 172.21.15.174:0/2106993017 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f55540263a0 con 0x7f55580f4740 2023-12-25T21:11:03.151 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.149+0000 7f554effd700 1 --2- 172.21.15.174:0/2106993017 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f554402ff60 0x7f5544032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:03.151 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.149+0000 7f554effd700 1 -- 172.21.15.174:0/2106993017 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f55540410d0 con 0x7f55580f4740 2023-12-25T21:11:03.151 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.149+0000 7f55605d6700 1 -- 172.21.15.174:0/2106993017 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5558059100 con 0x7f55580f4740 2023-12-25T21:11:03.151 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.149+0000 7f555db71700 1 --2- 172.21.15.174:0/2106993017 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f554402ff60 0x7f5544032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:03.152 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.150+0000 7f555db71700 1 --2- 172.21.15.174:0/2106993017 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f554402ff60 0x7f5544032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f55480096a0 tx=0x7f5548006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:03.155 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.154+0000 7f554effd700 1 -- 172.21.15.174:0/2106993017 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5554012920 con 0x7f55580f4740 2023-12-25T21:11:03.362 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.361+0000 7f55605d6700 1 -- 172.21.15.174:0/2106993017 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5558045ad0 con 0x7f55580f4740 2023-12-25T21:11:03.363 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.362+0000 7f554effd700 1 -- 172.21.15.174:0/2106993017 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f555402b020 con 0x7f55580f4740 2023-12-25T21:11:03.363 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:03.363 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:03.365 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.364+0000 7f55605d6700 1 -- 172.21.15.174:0/2106993017 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f554402ff60 msgr2=0x7f5544032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:03.365 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.364+0000 7f55605d6700 1 --2- 172.21.15.174:0/2106993017 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f554402ff60 0x7f5544032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f55480096a0 tx=0x7f5548006b40).stop 2023-12-25T21:11:03.365 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.364+0000 7f55605d6700 1 -- 172.21.15.174:0/2106993017 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55580f4740 msgr2=0x7f55581066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:03.365 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.364+0000 7f55605d6700 1 --2- 172.21.15.174:0/2106993017 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55580f4740 0x7f55581066b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f5554013510 tx=0x7f555400b040).stop 2023-12-25T21:11:03.365 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.364+0000 7f55605d6700 1 -- 172.21.15.174:0/2106993017 shutdown_connections 2023-12-25T21:11:03.365 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.364+0000 7f55605d6700 1 --2- 172.21.15.174:0/2106993017 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f554402ff60 0x7f5544032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:03.365 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.364+0000 7f55605d6700 1 --2- 172.21.15.174:0/2106993017 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55580f4740 0x7f55581066b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:03.365 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.364+0000 7f55605d6700 1 -- 172.21.15.174:0/2106993017 >> 172.21.15.174:0/2106993017 conn(0x7f55580f0260 msgr2=0x7f55580f0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:03.365 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.364+0000 7f55605d6700 1 -- 172.21.15.174:0/2106993017 shutdown_connections 2023-12-25T21:11:03.365 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:03.364+0000 7f55605d6700 1 -- 172.21.15.174:0/2106993017 wait complete. 2023-12-25T21:11:03.366 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:04.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:04 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:02.605881+0000 mgr.smithi061.jxxeke (mgr.14164) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:04.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:04 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:03.363503+0000 mon.smithi061 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.174:0/2106993017' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:04.753 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:04.753 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:05.871 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.868+0000 7f52e8b10700 1 -- 172.21.15.174:0/43931810 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52e40f5410 msgr2=0x7f52e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:05.871 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.868+0000 7f52e8b10700 1 --2- 172.21.15.174:0/43931810 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52e40f5410 0x7f52e40f5830 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f52cc009fb0 tx=0x7f52cc009b30).stop 2023-12-25T21:11:05.871 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.868+0000 7f52e8b10700 1 -- 172.21.15.174:0/43931810 shutdown_connections 2023-12-25T21:11:05.871 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.868+0000 7f52e8b10700 1 --2- 172.21.15.174:0/43931810 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52e40f5410 0x7f52e40f5830 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:05.871 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.868+0000 7f52e8b10700 1 -- 172.21.15.174:0/43931810 >> 172.21.15.174:0/43931810 conn(0x7f52e40f0b70 msgr2=0x7f52e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:05.871 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.868+0000 7f52e8b10700 1 -- 172.21.15.174:0/43931810 shutdown_connections 2023-12-25T21:11:05.871 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.868+0000 7f52e8b10700 1 -- 172.21.15.174:0/43931810 wait complete. 2023-12-25T21:11:05.871 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.869+0000 7f52e8b10700 1 Processor -- start 2023-12-25T21:11:05.871 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.869+0000 7f52e8b10700 1 -- start start 2023-12-25T21:11:05.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.869+0000 7f52e8b10700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52e40f5410 0x7f52e4106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:05.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.869+0000 7f52e8b10700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e4106c10 con 0x7f52e40f5410 2023-12-25T21:11:05.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.869+0000 7f52e259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52e40f5410 0x7f52e4106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:05.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.869+0000 7f52e259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52e40f5410 0x7f52e4106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:51374/0 (socket says 172.21.15.174:51374) 2023-12-25T21:11:05.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.869+0000 7f52e259c700 1 -- 172.21.15.174:0/825450030 learned_addr learned my addr 172.21.15.174:0/825450030 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:05.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.869+0000 7f52e259c700 1 -- 172.21.15.174:0/825450030 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52cc005040 con 0x7f52e40f5410 2023-12-25T21:11:05.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.870+0000 7f52e259c700 1 --2- 172.21.15.174:0/825450030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52e40f5410 0x7f52e4106720 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f52cc013510 tx=0x7f52cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:05.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.870+0000 7f52daffd700 1 -- 172.21.15.174:0/825450030 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52cc009320 con 0x7f52e40f5410 2023-12-25T21:11:05.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.870+0000 7f52e8b10700 1 -- 172.21.15.174:0/825450030 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e4106e10 con 0x7f52e40f5410 2023-12-25T21:11:05.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.870+0000 7f52daffd700 1 -- 172.21.15.174:0/825450030 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52cc013f00 con 0x7f52e40f5410 2023-12-25T21:11:05.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.870+0000 7f52daffd700 1 -- 172.21.15.174:0/825450030 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52cc014d90 con 0x7f52e40f5410 2023-12-25T21:11:05.873 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.870+0000 7f52e8b10700 1 -- 172.21.15.174:0/825450030 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e41072a0 con 0x7f52e40f5410 2023-12-25T21:11:05.873 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.871+0000 7f52daffd700 1 -- 172.21.15.174:0/825450030 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f52cc014420 con 0x7f52e40f5410 2023-12-25T21:11:05.874 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.871+0000 7f52e8b10700 1 -- 172.21.15.174:0/825450030 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e4045ad0 con 0x7f52e40f5410 2023-12-25T21:11:05.874 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.871+0000 7f52daffd700 1 --2- 172.21.15.174:0/825450030 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f52d002ff60 0x7f52d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:05.874 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.871+0000 7f52daffd700 1 -- 172.21.15.174:0/825450030 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f52cc040ff0 con 0x7f52e40f5410 2023-12-25T21:11:05.874 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.872+0000 7f52e1d9b700 1 --2- 172.21.15.174:0/825450030 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f52d002ff60 0x7f52d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:05.874 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.872+0000 7f52e1d9b700 1 --2- 172.21.15.174:0/825450030 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f52d002ff60 0x7f52d0032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f52d40096a0 tx=0x7f52d4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:05.877 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:05.876+0000 7f52daffd700 1 -- 172.21.15.174:0/825450030 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f52cc0095f0 con 0x7f52e40f5410 2023-12-25T21:11:06.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:06.082+0000 7f52e8b10700 1 -- 172.21.15.174:0/825450030 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f52e4108150 con 0x7f52e40f5410 2023-12-25T21:11:06.084 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:06.083+0000 7f52daffd700 1 -- 172.21.15.174:0/825450030 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f52cc01db70 con 0x7f52e40f5410 2023-12-25T21:11:06.084 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:06.084 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:06.085 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:06.085+0000 7f52e8b10700 1 -- 172.21.15.174:0/825450030 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f52d002ff60 msgr2=0x7f52d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:06.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:06.085+0000 7f52e8b10700 1 --2- 172.21.15.174:0/825450030 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f52d002ff60 0x7f52d0032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f52d40096a0 tx=0x7f52d4006b40).stop 2023-12-25T21:11:06.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:06.085+0000 7f52e8b10700 1 -- 172.21.15.174:0/825450030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52e40f5410 msgr2=0x7f52e4106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:06.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:06.085+0000 7f52e8b10700 1 --2- 172.21.15.174:0/825450030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52e40f5410 0x7f52e4106720 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f52cc013510 tx=0x7f52cc00b040).stop 2023-12-25T21:11:06.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:06.085+0000 7f52e8b10700 1 -- 172.21.15.174:0/825450030 shutdown_connections 2023-12-25T21:11:06.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:06.085+0000 7f52e8b10700 1 --2- 172.21.15.174:0/825450030 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f52d002ff60 0x7f52d0032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:06.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:06.085+0000 7f52e8b10700 1 --2- 172.21.15.174:0/825450030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52e40f5410 0x7f52e4106720 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:06.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:06.085+0000 7f52e8b10700 1 -- 172.21.15.174:0/825450030 >> 172.21.15.174:0/825450030 conn(0x7f52e40f0b70 msgr2=0x7f52e405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:06.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:06.085+0000 7f52e8b10700 1 -- 172.21.15.174:0/825450030 shutdown_connections 2023-12-25T21:11:06.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:06.085+0000 7f52e8b10700 1 -- 172.21.15.174:0/825450030 wait complete. 2023-12-25T21:11:06.086 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:06.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:06 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:04.606216+0000 mgr.smithi061.jxxeke (mgr.14164) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:06.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:06 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:06.084606+0000 mon.smithi061 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.174:0/825450030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:07.483 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:07.484 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:08.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:08 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:06.606516+0000 mgr.smithi061.jxxeke (mgr.14164) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:08.709 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.708+0000 7fe974bae700 1 -- 172.21.15.174:0/3333166430 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9700f4740 msgr2=0x7fe9700f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:08.709 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.708+0000 7fe974bae700 1 --2- 172.21.15.174:0/3333166430 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9700f4740 0x7fe9700f4b60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fe958009fb0 tx=0x7fe958009b30).stop 2023-12-25T21:11:08.709 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.708+0000 7fe974bae700 1 -- 172.21.15.174:0/3333166430 shutdown_connections 2023-12-25T21:11:08.709 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.708+0000 7fe974bae700 1 --2- 172.21.15.174:0/3333166430 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9700f4740 0x7fe9700f4b60 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:08.709 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.708+0000 7fe974bae700 1 -- 172.21.15.174:0/3333166430 >> 172.21.15.174:0/3333166430 conn(0x7fe9700f0260 msgr2=0x7fe9700f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:08.709 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.708+0000 7fe974bae700 1 -- 172.21.15.174:0/3333166430 shutdown_connections 2023-12-25T21:11:08.709 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.708+0000 7fe974bae700 1 -- 172.21.15.174:0/3333166430 wait complete. 2023-12-25T21:11:08.709 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.708+0000 7fe974bae700 1 Processor -- start 2023-12-25T21:11:08.709 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.708+0000 7fe974bae700 1 -- start start 2023-12-25T21:11:08.709 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.708+0000 7fe974bae700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9700f4740 0x7fe9701066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:08.710 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.708+0000 7fe974bae700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe970106bc0 con 0x7fe9700f4740 2023-12-25T21:11:08.711 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.708+0000 7fe96e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9700f4740 0x7fe9701066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:08.711 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.709+0000 7fe96e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9700f4740 0x7fe9701066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:44290/0 (socket says 172.21.15.174:44290) 2023-12-25T21:11:08.711 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.709+0000 7fe96e59c700 1 -- 172.21.15.174:0/2153931896 learned_addr learned my addr 172.21.15.174:0/2153931896 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:08.711 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.709+0000 7fe96e59c700 1 -- 172.21.15.174:0/2153931896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe958005040 con 0x7fe9700f4740 2023-12-25T21:11:08.711 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.709+0000 7fe96e59c700 1 --2- 172.21.15.174:0/2153931896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9700f4740 0x7fe9701066d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fe958013510 tx=0x7fe95800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:08.711 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.709+0000 7fe966ffd700 1 -- 172.21.15.174:0/2153931896 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe958009320 con 0x7fe9700f4740 2023-12-25T21:11:08.711 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.709+0000 7fe974bae700 1 -- 172.21.15.174:0/2153931896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe970106dc0 con 0x7fe9700f4740 2023-12-25T21:11:08.712 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.709+0000 7fe966ffd700 1 -- 172.21.15.174:0/2153931896 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe958013f00 con 0x7fe9700f4740 2023-12-25T21:11:08.712 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.709+0000 7fe966ffd700 1 -- 172.21.15.174:0/2153931896 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe958014d90 con 0x7fe9700f4740 2023-12-25T21:11:08.712 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.709+0000 7fe974bae700 1 -- 172.21.15.174:0/2153931896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe970107250 con 0x7fe9700f4740 2023-12-25T21:11:08.712 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.710+0000 7fe974bae700 1 -- 172.21.15.174:0/2153931896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe970045ad0 con 0x7fe9700f4740 2023-12-25T21:11:08.712 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.710+0000 7fe966ffd700 1 -- 172.21.15.174:0/2153931896 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9580095f0 con 0x7fe9700f4740 2023-12-25T21:11:08.712 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.710+0000 7fe966ffd700 1 --2- 172.21.15.174:0/2153931896 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe95c02ffb0 0x7fe95c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:08.712 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.710+0000 7fe966ffd700 1 -- 172.21.15.174:0/2153931896 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe958041080 con 0x7fe9700f4740 2023-12-25T21:11:08.712 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.710+0000 7fe96dd9b700 1 --2- 172.21.15.174:0/2153931896 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe95c02ffb0 0x7fe95c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:08.712 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.711+0000 7fe96dd9b700 1 --2- 172.21.15.174:0/2153931896 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe95c02ffb0 0x7fe95c032470 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fe9600096a0 tx=0x7fe960006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:08.716 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.715+0000 7fe966ffd700 1 -- 172.21.15.174:0/2153931896 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe95806d2b0 con 0x7fe9700f4740 2023-12-25T21:11:08.922 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.921+0000 7fe974bae700 1 -- 172.21.15.174:0/2153931896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe970108100 con 0x7fe9700f4740 2023-12-25T21:11:08.922 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.921+0000 7fe966ffd700 1 -- 172.21.15.174:0/2153931896 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe958014420 con 0x7fe9700f4740 2023-12-25T21:11:08.922 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:08.922 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:08.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.923+0000 7fe974bae700 1 -- 172.21.15.174:0/2153931896 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe95c02ffb0 msgr2=0x7fe95c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:08.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.923+0000 7fe974bae700 1 --2- 172.21.15.174:0/2153931896 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe95c02ffb0 0x7fe95c032470 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fe9600096a0 tx=0x7fe960006b40).stop 2023-12-25T21:11:08.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.923+0000 7fe974bae700 1 -- 172.21.15.174:0/2153931896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9700f4740 msgr2=0x7fe9701066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:08.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.923+0000 7fe974bae700 1 --2- 172.21.15.174:0/2153931896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9700f4740 0x7fe9701066d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fe958013510 tx=0x7fe95800b040).stop 2023-12-25T21:11:08.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.923+0000 7fe974bae700 1 -- 172.21.15.174:0/2153931896 shutdown_connections 2023-12-25T21:11:08.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.923+0000 7fe974bae700 1 --2- 172.21.15.174:0/2153931896 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe95c02ffb0 0x7fe95c032470 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:08.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.923+0000 7fe974bae700 1 --2- 172.21.15.174:0/2153931896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9700f4740 0x7fe9701066d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:08.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.923+0000 7fe974bae700 1 -- 172.21.15.174:0/2153931896 >> 172.21.15.174:0/2153931896 conn(0x7fe9700f0260 msgr2=0x7fe9700f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:08.924 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.923+0000 7fe974bae700 1 -- 172.21.15.174:0/2153931896 shutdown_connections 2023-12-25T21:11:08.925 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:08.923+0000 7fe974bae700 1 -- 172.21.15.174:0/2153931896 wait complete. 2023-12-25T21:11:08.925 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:09.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:09 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:08.922708+0000 mon.smithi061 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.174:0/2153931896' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:10.326 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:10.327 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:10.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:10 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:08.606843+0000 mgr.smithi061.jxxeke (mgr.14164) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:11.518 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.516+0000 7f4103df7700 1 -- 172.21.15.174:0/3658633263 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40fc060220 msgr2=0x7f40fc0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:11.519 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.516+0000 7f4103df7700 1 --2- 172.21.15.174:0/3658633263 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40fc060220 0x7f40fc0f9150 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f40ec009fb0 tx=0x7f40ec009b30).stop 2023-12-25T21:11:11.519 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.516+0000 7f4103df7700 1 -- 172.21.15.174:0/3658633263 shutdown_connections 2023-12-25T21:11:11.519 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.516+0000 7f4103df7700 1 --2- 172.21.15.174:0/3658633263 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40fc060220 0x7f40fc0f9150 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:11.519 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.516+0000 7f4103df7700 1 -- 172.21.15.174:0/3658633263 >> 172.21.15.174:0/3658633263 conn(0x7f40fc0f0b70 msgr2=0x7f40fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:11.519 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.516+0000 7f4103df7700 1 -- 172.21.15.174:0/3658633263 shutdown_connections 2023-12-25T21:11:11.519 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.516+0000 7f4103df7700 1 -- 172.21.15.174:0/3658633263 wait complete. 2023-12-25T21:11:11.519 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.517+0000 7f4103df7700 1 Processor -- start 2023-12-25T21:11:11.519 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.517+0000 7f4103df7700 1 -- start start 2023-12-25T21:11:11.519 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.517+0000 7f4103df7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40fc060220 0x7f40fc1066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:11.519 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.517+0000 7f4103df7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc106b90 con 0x7f40fc060220 2023-12-25T21:11:11.519 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.517+0000 7f4101b93700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40fc060220 0x7f40fc1066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:11.519 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.517+0000 7f4101b93700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40fc060220 0x7f40fc1066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:44322/0 (socket says 172.21.15.174:44322) 2023-12-25T21:11:11.519 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.517+0000 7f4101b93700 1 -- 172.21.15.174:0/2514536267 learned_addr learned my addr 172.21.15.174:0/2514536267 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:11.519 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.517+0000 7f4101b93700 1 -- 172.21.15.174:0/2514536267 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ec005040 con 0x7f40fc060220 2023-12-25T21:11:11.520 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.517+0000 7f4101b93700 1 --2- 172.21.15.174:0/2514536267 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40fc060220 0x7f40fc1066a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f40ec013510 tx=0x7f40ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:11.520 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.518+0000 7f40f27fc700 1 -- 172.21.15.174:0/2514536267 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40ec009320 con 0x7f40fc060220 2023-12-25T21:11:11.520 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.518+0000 7f4103df7700 1 -- 172.21.15.174:0/2514536267 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40fc106d90 con 0x7f40fc060220 2023-12-25T21:11:11.520 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.518+0000 7f40f27fc700 1 -- 172.21.15.174:0/2514536267 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40ec013f00 con 0x7f40fc060220 2023-12-25T21:11:11.520 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.518+0000 7f40f27fc700 1 -- 172.21.15.174:0/2514536267 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40ec014dc0 con 0x7f40fc060220 2023-12-25T21:11:11.520 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.518+0000 7f4103df7700 1 -- 172.21.15.174:0/2514536267 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40fc107220 con 0x7f40fc060220 2023-12-25T21:11:11.521 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.518+0000 7f40f27fc700 1 -- 172.21.15.174:0/2514536267 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40ec0095f0 con 0x7f40fc060220 2023-12-25T21:11:11.521 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.518+0000 7f4103df7700 1 -- 172.21.15.174:0/2514536267 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40fc045ad0 con 0x7f40fc060220 2023-12-25T21:11:11.521 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.519+0000 7f40f27fc700 1 --2- 172.21.15.174:0/2514536267 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f40e802ff60 0x7f40e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:11.521 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.519+0000 7f40f27fc700 1 -- 172.21.15.174:0/2514536267 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f40ec041000 con 0x7f40fc060220 2023-12-25T21:11:11.521 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.519+0000 7f4101392700 1 --2- 172.21.15.174:0/2514536267 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f40e802ff60 0x7f40e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:11.521 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.519+0000 7f4101392700 1 --2- 172.21.15.174:0/2514536267 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f40e802ff60 0x7f40e8032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f40f80096a0 tx=0x7f40f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:11.524 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.524+0000 7f40f27fc700 1 -- 172.21.15.174:0/2514536267 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40ec014420 con 0x7f40fc060220 2023-12-25T21:11:11.729 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.729+0000 7f4103df7700 1 -- 172.21.15.174:0/2514536267 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40fc1080d0 con 0x7f40fc060220 2023-12-25T21:11:11.730 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.729+0000 7f40f27fc700 1 -- 172.21.15.174:0/2514536267 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f40ec010230 con 0x7f40fc060220 2023-12-25T21:11:11.730 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:11.731 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:11.732 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.731+0000 7f4103df7700 1 -- 172.21.15.174:0/2514536267 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f40e802ff60 msgr2=0x7f40e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:11.732 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.731+0000 7f4103df7700 1 --2- 172.21.15.174:0/2514536267 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f40e802ff60 0x7f40e8032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f40f80096a0 tx=0x7f40f8006b40).stop 2023-12-25T21:11:11.732 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.731+0000 7f4103df7700 1 -- 172.21.15.174:0/2514536267 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40fc060220 msgr2=0x7f40fc1066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:11.732 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.731+0000 7f4103df7700 1 --2- 172.21.15.174:0/2514536267 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40fc060220 0x7f40fc1066a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f40ec013510 tx=0x7f40ec00b040).stop 2023-12-25T21:11:11.732 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.731+0000 7f4103df7700 1 -- 172.21.15.174:0/2514536267 shutdown_connections 2023-12-25T21:11:11.732 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.731+0000 7f4103df7700 1 --2- 172.21.15.174:0/2514536267 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f40e802ff60 0x7f40e8032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:11.732 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.731+0000 7f4103df7700 1 --2- 172.21.15.174:0/2514536267 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40fc060220 0x7f40fc1066a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:11.732 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.731+0000 7f4103df7700 1 -- 172.21.15.174:0/2514536267 >> 172.21.15.174:0/2514536267 conn(0x7f40fc0f0b70 msgr2=0x7f40fc0f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:11.732 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.731+0000 7f4103df7700 1 -- 172.21.15.174:0/2514536267 shutdown_connections 2023-12-25T21:11:11.732 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:11.731+0000 7f4103df7700 1 -- 172.21.15.174:0/2514536267 wait complete. 2023-12-25T21:11:11.733 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:12.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:12 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:10.607220+0000 mgr.smithi061.jxxeke (mgr.14164) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:12.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:12 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:11.730719+0000 mon.smithi061 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.174:0/2514536267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:13.129 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:13.129 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:13.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:13 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:12.607509+0000 mgr.smithi061.jxxeke (mgr.14164) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:14.335 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.331+0000 7fdc90383700 1 -- 172.21.15.174:0/1556640846 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc88060280 msgr2=0x7fdc880f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:14.335 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.331+0000 7fdc90383700 1 --2- 172.21.15.174:0/1556640846 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc88060280 0x7fdc880f90f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fdc84009fb0 tx=0x7fdc84009b30).stop 2023-12-25T21:11:14.335 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.332+0000 7fdc90383700 1 -- 172.21.15.174:0/1556640846 shutdown_connections 2023-12-25T21:11:14.335 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.332+0000 7fdc90383700 1 --2- 172.21.15.174:0/1556640846 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc88060280 0x7fdc880f90f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:14.335 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.332+0000 7fdc90383700 1 -- 172.21.15.174:0/1556640846 >> 172.21.15.174:0/1556640846 conn(0x7fdc880f0b70 msgr2=0x7fdc880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:14.335 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.332+0000 7fdc90383700 1 -- 172.21.15.174:0/1556640846 shutdown_connections 2023-12-25T21:11:14.336 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.332+0000 7fdc90383700 1 -- 172.21.15.174:0/1556640846 wait complete. 2023-12-25T21:11:14.336 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.332+0000 7fdc90383700 1 Processor -- start 2023-12-25T21:11:14.336 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.332+0000 7fdc90383700 1 -- start start 2023-12-25T21:11:14.336 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.332+0000 7fdc90383700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc88060280 0x7fdc880f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:14.336 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.332+0000 7fdc90383700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc880f4ff0 con 0x7fdc88060280 2023-12-25T21:11:14.336 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.332+0000 7fdc8e11f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc88060280 0x7fdc880f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:14.336 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.332+0000 7fdc8e11f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc88060280 0x7fdc880f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:44350/0 (socket says 172.21.15.174:44350) 2023-12-25T21:11:14.336 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.332+0000 7fdc8e11f700 1 -- 172.21.15.174:0/1014833040 learned_addr learned my addr 172.21.15.174:0/1014833040 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:14.336 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.333+0000 7fdc8e11f700 1 -- 172.21.15.174:0/1014833040 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc84005040 con 0x7fdc88060280 2023-12-25T21:11:14.336 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.333+0000 7fdc8e11f700 1 --2- 172.21.15.174:0/1014833040 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc88060280 0x7fdc880f6b70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fdc84013510 tx=0x7fdc8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:14.336 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.333+0000 7fdc7effd700 1 -- 172.21.15.174:0/1014833040 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc84009320 con 0x7fdc88060280 2023-12-25T21:11:14.336 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.333+0000 7fdc90383700 1 -- 172.21.15.174:0/1014833040 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc880f5190 con 0x7fdc88060280 2023-12-25T21:11:14.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.333+0000 7fdc7effd700 1 -- 172.21.15.174:0/1014833040 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc84013f00 con 0x7fdc88060280 2023-12-25T21:11:14.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.333+0000 7fdc7effd700 1 -- 172.21.15.174:0/1014833040 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc84014dc0 con 0x7fdc88060280 2023-12-25T21:11:14.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.333+0000 7fdc90383700 1 -- 172.21.15.174:0/1014833040 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc880f5620 con 0x7fdc88060280 2023-12-25T21:11:14.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.334+0000 7fdc7effd700 1 -- 172.21.15.174:0/1014833040 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc840095f0 con 0x7fdc88060280 2023-12-25T21:11:14.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.334+0000 7fdc90383700 1 -- 172.21.15.174:0/1014833040 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc88059100 con 0x7fdc88060280 2023-12-25T21:11:14.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.334+0000 7fdc7effd700 1 --2- 172.21.15.174:0/1014833040 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdc7402ff60 0x7fdc74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:14.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.334+0000 7fdc7effd700 1 -- 172.21.15.174:0/1014833040 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc840410c0 con 0x7fdc88060280 2023-12-25T21:11:14.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.334+0000 7fdc8d91e700 1 --2- 172.21.15.174:0/1014833040 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdc7402ff60 0x7fdc74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:14.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.335+0000 7fdc8d91e700 1 --2- 172.21.15.174:0/1014833040 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdc7402ff60 0x7fdc74032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fdc780096a0 tx=0x7fdc78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:14.339 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.339+0000 7fdc7effd700 1 -- 172.21.15.174:0/1014833040 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc84014420 con 0x7fdc88060280 2023-12-25T21:11:14.543 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.542+0000 7fdc90383700 1 -- 172.21.15.174:0/1014833040 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc880fbf50 con 0x7fdc88060280 2023-12-25T21:11:14.544 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.543+0000 7fdc7effd700 1 -- 172.21.15.174:0/1014833040 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdc84010230 con 0x7fdc88060280 2023-12-25T21:11:14.544 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:14.544 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:14.545 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.545+0000 7fdc90383700 1 -- 172.21.15.174:0/1014833040 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdc7402ff60 msgr2=0x7fdc74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:14.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.545+0000 7fdc90383700 1 --2- 172.21.15.174:0/1014833040 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdc7402ff60 0x7fdc74032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fdc780096a0 tx=0x7fdc78006b40).stop 2023-12-25T21:11:14.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.545+0000 7fdc90383700 1 -- 172.21.15.174:0/1014833040 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc88060280 msgr2=0x7fdc880f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:14.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.545+0000 7fdc90383700 1 --2- 172.21.15.174:0/1014833040 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc88060280 0x7fdc880f6b70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fdc84013510 tx=0x7fdc8400b040).stop 2023-12-25T21:11:14.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.545+0000 7fdc90383700 1 -- 172.21.15.174:0/1014833040 shutdown_connections 2023-12-25T21:11:14.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.545+0000 7fdc90383700 1 --2- 172.21.15.174:0/1014833040 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdc7402ff60 0x7fdc74032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:14.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.545+0000 7fdc90383700 1 --2- 172.21.15.174:0/1014833040 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc88060280 0x7fdc880f6b70 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:14.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.545+0000 7fdc90383700 1 -- 172.21.15.174:0/1014833040 >> 172.21.15.174:0/1014833040 conn(0x7fdc880f0b70 msgr2=0x7fdc880f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:14.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.545+0000 7fdc90383700 1 -- 172.21.15.174:0/1014833040 shutdown_connections 2023-12-25T21:11:14.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:14.545+0000 7fdc90383700 1 -- 172.21.15.174:0/1014833040 wait complete. 2023-12-25T21:11:14.547 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:14.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:14 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:14.544624+0000 mon.smithi061 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.174:0/1014833040' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:15.934 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:15.934 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:15.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:15 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:14.607830+0000 mgr.smithi061.jxxeke (mgr.14164) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:17.058 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.056+0000 7f8795a86700 1 -- 172.21.15.174:0/4102020124 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87900f6510 msgr2=0x7f87900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:17.058 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.056+0000 7f8795a86700 1 --2- 172.21.15.174:0/4102020124 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87900f6510 0x7f87900f6930 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f8784009fb0 tx=0x7f8784009b30).stop 2023-12-25T21:11:17.058 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.056+0000 7f8795a86700 1 -- 172.21.15.174:0/4102020124 shutdown_connections 2023-12-25T21:11:17.058 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.056+0000 7f8795a86700 1 --2- 172.21.15.174:0/4102020124 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87900f6510 0x7f87900f6930 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:17.058 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.056+0000 7f8795a86700 1 -- 172.21.15.174:0/4102020124 >> 172.21.15.174:0/4102020124 conn(0x7f87900f1c70 msgr2=0x7f87900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:17.058 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.056+0000 7f8795a86700 1 -- 172.21.15.174:0/4102020124 shutdown_connections 2023-12-25T21:11:17.058 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.056+0000 7f8795a86700 1 -- 172.21.15.174:0/4102020124 wait complete. 2023-12-25T21:11:17.058 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.057+0000 7f8795a86700 1 Processor -- start 2023-12-25T21:11:17.059 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.057+0000 7f8795a86700 1 -- start start 2023-12-25T21:11:17.059 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.057+0000 7f8795a86700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87900f6510 0x7f879006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:17.059 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.057+0000 7f8795a86700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8790069e20 con 0x7f87900f6510 2023-12-25T21:11:17.059 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.057+0000 7f878f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87900f6510 0x7f879006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:17.059 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.057+0000 7f878f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87900f6510 0x7f879006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:46624/0 (socket says 172.21.15.174:46624) 2023-12-25T21:11:17.059 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.057+0000 7f878f7fe700 1 -- 172.21.15.174:0/764171322 learned_addr learned my addr 172.21.15.174:0/764171322 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:17.060 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.058+0000 7f878f7fe700 1 -- 172.21.15.174:0/764171322 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8784005040 con 0x7f87900f6510 2023-12-25T21:11:17.060 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.058+0000 7f878f7fe700 1 --2- 172.21.15.174:0/764171322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87900f6510 0x7f879006b9a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f8784013510 tx=0x7f878400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:17.060 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.058+0000 7f8777fff700 1 -- 172.21.15.174:0/764171322 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8784009320 con 0x7f87900f6510 2023-12-25T21:11:17.061 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.058+0000 7f8795a86700 1 -- 172.21.15.174:0/764171322 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8790069fc0 con 0x7f87900f6510 2023-12-25T21:11:17.061 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.058+0000 7f8777fff700 1 -- 172.21.15.174:0/764171322 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8784013f00 con 0x7f87900f6510 2023-12-25T21:11:17.061 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.058+0000 7f8777fff700 1 -- 172.21.15.174:0/764171322 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8784014d90 con 0x7f87900f6510 2023-12-25T21:11:17.061 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.058+0000 7f8795a86700 1 -- 172.21.15.174:0/764171322 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f879006a450 con 0x7f87900f6510 2023-12-25T21:11:17.061 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.059+0000 7f8777fff700 1 -- 172.21.15.174:0/764171322 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8784014420 con 0x7f87900f6510 2023-12-25T21:11:17.061 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.059+0000 7f8795a86700 1 -- 172.21.15.174:0/764171322 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8790059100 con 0x7f87900f6510 2023-12-25T21:11:17.061 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.059+0000 7f8777fff700 1 --2- 172.21.15.174:0/764171322 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f877802ff60 0x7f8778032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:17.061 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.059+0000 7f8777fff700 1 -- 172.21.15.174:0/764171322 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f87840410c0 con 0x7f87900f6510 2023-12-25T21:11:17.061 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.059+0000 7f878effd700 1 --2- 172.21.15.174:0/764171322 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f877802ff60 0x7f8778032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:17.061 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.059+0000 7f878effd700 1 --2- 172.21.15.174:0/764171322 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f877802ff60 0x7f8778032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f87800096a0 tx=0x7f8780006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:17.064 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.065+0000 7f8777fff700 1 -- 172.21.15.174:0/764171322 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87840095f0 con 0x7f87900f6510 2023-12-25T21:11:17.269 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.268+0000 7f8795a86700 1 -- 172.21.15.174:0/764171322 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87900fbf60 con 0x7f87900f6510 2023-12-25T21:11:17.270 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.270+0000 7f8777fff700 1 -- 172.21.15.174:0/764171322 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f878401db70 con 0x7f87900f6510 2023-12-25T21:11:17.270 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:17.270 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:17.271 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.271+0000 7f8795a86700 1 -- 172.21.15.174:0/764171322 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f877802ff60 msgr2=0x7f8778032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:17.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.271+0000 7f8795a86700 1 --2- 172.21.15.174:0/764171322 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f877802ff60 0x7f8778032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f87800096a0 tx=0x7f8780006b40).stop 2023-12-25T21:11:17.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.271+0000 7f8795a86700 1 -- 172.21.15.174:0/764171322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87900f6510 msgr2=0x7f879006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:17.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.271+0000 7f8795a86700 1 --2- 172.21.15.174:0/764171322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87900f6510 0x7f879006b9a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f8784013510 tx=0x7f878400b040).stop 2023-12-25T21:11:17.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.271+0000 7f8795a86700 1 -- 172.21.15.174:0/764171322 shutdown_connections 2023-12-25T21:11:17.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.271+0000 7f8795a86700 1 --2- 172.21.15.174:0/764171322 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f877802ff60 0x7f8778032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:17.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.271+0000 7f8795a86700 1 --2- 172.21.15.174:0/764171322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87900f6510 0x7f879006b9a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:17.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.271+0000 7f8795a86700 1 -- 172.21.15.174:0/764171322 >> 172.21.15.174:0/764171322 conn(0x7f87900f1c70 msgr2=0x7f87900f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:17.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.271+0000 7f8795a86700 1 -- 172.21.15.174:0/764171322 shutdown_connections 2023-12-25T21:11:17.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:17.271+0000 7f8795a86700 1 -- 172.21.15.174:0/764171322 wait complete. 2023-12-25T21:11:17.273 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:17.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:17 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:17.270398+0000 mon.smithi061 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.174:0/764171322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:18.659 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:18.659 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:18.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:18 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:16.608095+0000 mgr.smithi061.jxxeke (mgr.14164) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:19.794 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.793+0000 7fdc6b1df700 1 -- 172.21.15.174:0/976949696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc640f5410 msgr2=0x7fdc640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:19.794 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.793+0000 7fdc6b1df700 1 --2- 172.21.15.174:0/976949696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc640f5410 0x7fdc640f5830 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fdc58009fb0 tx=0x7fdc58009b30).stop 2023-12-25T21:11:19.794 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.793+0000 7fdc6b1df700 1 -- 172.21.15.174:0/976949696 shutdown_connections 2023-12-25T21:11:19.794 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.793+0000 7fdc6b1df700 1 --2- 172.21.15.174:0/976949696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc640f5410 0x7fdc640f5830 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:19.794 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.793+0000 7fdc6b1df700 1 -- 172.21.15.174:0/976949696 >> 172.21.15.174:0/976949696 conn(0x7fdc640f0b70 msgr2=0x7fdc640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:19.794 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.793+0000 7fdc6b1df700 1 -- 172.21.15.174:0/976949696 shutdown_connections 2023-12-25T21:11:19.794 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.793+0000 7fdc6b1df700 1 -- 172.21.15.174:0/976949696 wait complete. 2023-12-25T21:11:19.794 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.794+0000 7fdc6b1df700 1 Processor -- start 2023-12-25T21:11:19.794 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.794+0000 7fdc6b1df700 1 -- start start 2023-12-25T21:11:19.794 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.794+0000 7fdc6b1df700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc640f5410 0x7fdc64106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:19.794 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.794+0000 7fdc6b1df700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc64106c10 con 0x7fdc640f5410 2023-12-25T21:11:19.795 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.794+0000 7fdc68f7b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc640f5410 0x7fdc64106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:19.795 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.794+0000 7fdc68f7b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc640f5410 0x7fdc64106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:46646/0 (socket says 172.21.15.174:46646) 2023-12-25T21:11:19.795 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.794+0000 7fdc68f7b700 1 -- 172.21.15.174:0/646191090 learned_addr learned my addr 172.21.15.174:0/646191090 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:19.796 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.794+0000 7fdc68f7b700 1 -- 172.21.15.174:0/646191090 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc58005040 con 0x7fdc640f5410 2023-12-25T21:11:19.796 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.795+0000 7fdc68f7b700 1 --2- 172.21.15.174:0/646191090 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc640f5410 0x7fdc64106720 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fdc58013510 tx=0x7fdc5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:19.796 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.795+0000 7fdc617fa700 1 -- 172.21.15.174:0/646191090 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc58009320 con 0x7fdc640f5410 2023-12-25T21:11:19.796 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.795+0000 7fdc6b1df700 1 -- 172.21.15.174:0/646191090 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc64106e10 con 0x7fdc640f5410 2023-12-25T21:11:19.796 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.795+0000 7fdc617fa700 1 -- 172.21.15.174:0/646191090 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc58013f00 con 0x7fdc640f5410 2023-12-25T21:11:19.797 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.795+0000 7fdc617fa700 1 -- 172.21.15.174:0/646191090 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc58014d90 con 0x7fdc640f5410 2023-12-25T21:11:19.797 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.795+0000 7fdc6b1df700 1 -- 172.21.15.174:0/646191090 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc641072a0 con 0x7fdc640f5410 2023-12-25T21:11:19.797 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.795+0000 7fdc617fa700 1 -- 172.21.15.174:0/646191090 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc580263a0 con 0x7fdc640f5410 2023-12-25T21:11:19.797 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.796+0000 7fdc617fa700 1 --2- 172.21.15.174:0/646191090 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdc5402ff60 0x7fdc54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:19.797 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.796+0000 7fdc617fa700 1 -- 172.21.15.174:0/646191090 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc580410d0 con 0x7fdc640f5410 2023-12-25T21:11:19.797 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.796+0000 7fdc6b1df700 1 -- 172.21.15.174:0/646191090 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc64059100 con 0x7fdc640f5410 2023-12-25T21:11:19.797 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.796+0000 7fdc63fff700 1 --2- 172.21.15.174:0/646191090 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdc5402ff60 0x7fdc54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:19.797 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.796+0000 7fdc63fff700 1 --2- 172.21.15.174:0/646191090 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdc5402ff60 0x7fdc54032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fdc500096a0 tx=0x7fdc50006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:19.800 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:19.801+0000 7fdc617fa700 1 -- 172.21.15.174:0/646191090 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc58012920 con 0x7fdc640f5410 2023-12-25T21:11:20.006 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:20.006+0000 7fdc6b1df700 1 -- 172.21.15.174:0/646191090 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc640fae60 con 0x7fdc640f5410 2023-12-25T21:11:20.007 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:20.007+0000 7fdc617fa700 1 -- 172.21.15.174:0/646191090 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdc5802b020 con 0x7fdc640f5410 2023-12-25T21:11:20.007 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:20.007 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:20.008 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:20.009+0000 7fdc6b1df700 1 -- 172.21.15.174:0/646191090 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdc5402ff60 msgr2=0x7fdc54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:20.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:20.009+0000 7fdc6b1df700 1 --2- 172.21.15.174:0/646191090 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdc5402ff60 0x7fdc54032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fdc500096a0 tx=0x7fdc50006b40).stop 2023-12-25T21:11:20.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:20.009+0000 7fdc6b1df700 1 -- 172.21.15.174:0/646191090 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc640f5410 msgr2=0x7fdc64106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:20.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:20.009+0000 7fdc6b1df700 1 --2- 172.21.15.174:0/646191090 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc640f5410 0x7fdc64106720 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fdc58013510 tx=0x7fdc5800b040).stop 2023-12-25T21:11:20.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:20.009+0000 7fdc6b1df700 1 -- 172.21.15.174:0/646191090 shutdown_connections 2023-12-25T21:11:20.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:20.009+0000 7fdc6b1df700 1 --2- 172.21.15.174:0/646191090 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdc5402ff60 0x7fdc54032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:20.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:20.009+0000 7fdc6b1df700 1 --2- 172.21.15.174:0/646191090 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc640f5410 0x7fdc64106720 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:20.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:20.009+0000 7fdc6b1df700 1 -- 172.21.15.174:0/646191090 >> 172.21.15.174:0/646191090 conn(0x7fdc640f0b70 msgr2=0x7fdc6405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:20.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:20.009+0000 7fdc6b1df700 1 -- 172.21.15.174:0/646191090 shutdown_connections 2023-12-25T21:11:20.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:20.009+0000 7fdc6b1df700 1 -- 172.21.15.174:0/646191090 wait complete. 2023-12-25T21:11:20.010 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:20.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:20 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:18.608418+0000 mgr.smithi061.jxxeke (mgr.14164) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:20.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:20 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:20.007451+0000 mon.smithi061 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.174:0/646191090' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:21.382 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:21.382 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:22.611 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.610+0000 7f130d528700 1 -- 172.21.15.174:0/2891430735 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13080f51a0 msgr2=0x7f13080f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:22.612 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.610+0000 7f130d528700 1 --2- 172.21.15.174:0/2891430735 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13080f51a0 0x7f13080f55c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f12f8009fb0 tx=0x7f12f8009b30).stop 2023-12-25T21:11:22.612 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.610+0000 7f130d528700 1 -- 172.21.15.174:0/2891430735 shutdown_connections 2023-12-25T21:11:22.612 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.610+0000 7f130d528700 1 --2- 172.21.15.174:0/2891430735 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13080f51a0 0x7f13080f55c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:22.612 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.610+0000 7f130d528700 1 -- 172.21.15.174:0/2891430735 >> 172.21.15.174:0/2891430735 conn(0x7f13080f09a0 msgr2=0x7f13080f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:22.612 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.610+0000 7f130d528700 1 -- 172.21.15.174:0/2891430735 shutdown_connections 2023-12-25T21:11:22.612 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.610+0000 7f130d528700 1 -- 172.21.15.174:0/2891430735 wait complete. 2023-12-25T21:11:22.612 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.611+0000 7f130d528700 1 Processor -- start 2023-12-25T21:11:22.612 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.611+0000 7f130d528700 1 -- start start 2023-12-25T21:11:22.612 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.612+0000 7f130d528700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13080f51a0 0x7f13081053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:22.612 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.612+0000 7f130d528700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13081058a0 con 0x7f13080f51a0 2023-12-25T21:11:22.612 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.612+0000 7f1307fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13080f51a0 0x7f13081053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:22.613 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.612+0000 7f1307fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13080f51a0 0x7f13081053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:46656/0 (socket says 172.21.15.174:46656) 2023-12-25T21:11:22.613 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.612+0000 7f1307fff700 1 -- 172.21.15.174:0/1316592361 learned_addr learned my addr 172.21.15.174:0/1316592361 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:22.614 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.612+0000 7f1307fff700 1 -- 172.21.15.174:0/1316592361 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f8005040 con 0x7f13080f51a0 2023-12-25T21:11:22.614 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.612+0000 7f1307fff700 1 --2- 172.21.15.174:0/1316592361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13080f51a0 0x7f13081053b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f12f8013510 tx=0x7f12f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:22.614 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.613+0000 7f1304ff9700 1 -- 172.21.15.174:0/1316592361 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12f8009320 con 0x7f13080f51a0 2023-12-25T21:11:22.614 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.613+0000 7f1304ff9700 1 -- 172.21.15.174:0/1316592361 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12f8013f00 con 0x7f13080f51a0 2023-12-25T21:11:22.614 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.613+0000 7f130d528700 1 -- 172.21.15.174:0/1316592361 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1308105aa0 con 0x7f13080f51a0 2023-12-25T21:11:22.615 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.613+0000 7f1304ff9700 1 -- 172.21.15.174:0/1316592361 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12f8014dc0 con 0x7f13080f51a0 2023-12-25T21:11:22.615 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.613+0000 7f130d528700 1 -- 172.21.15.174:0/1316592361 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1308105f30 con 0x7f13080f51a0 2023-12-25T21:11:22.615 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.614+0000 7f1304ff9700 1 -- 172.21.15.174:0/1316592361 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12f8014420 con 0x7f13080f51a0 2023-12-25T21:11:22.615 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.614+0000 7f130d528700 1 -- 172.21.15.174:0/1316592361 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1308045ad0 con 0x7f13080f51a0 2023-12-25T21:11:22.615 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.614+0000 7f1304ff9700 1 --2- 172.21.15.174:0/1316592361 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f12f002ffb0 0x7f12f0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:22.615 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.614+0000 7f1304ff9700 1 -- 172.21.15.174:0/1316592361 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12f8041110 con 0x7f13080f51a0 2023-12-25T21:11:22.615 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.614+0000 7f13077fe700 1 --2- 172.21.15.174:0/1316592361 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f12f002ffb0 0x7f12f0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:22.615 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.615+0000 7f13077fe700 1 --2- 172.21.15.174:0/1316592361 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f12f002ffb0 0x7f12f0032470 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f12fc0096a0 tx=0x7f12fc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:22.618 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.619+0000 7f1304ff9700 1 -- 172.21.15.174:0/1316592361 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12f80095f0 con 0x7f13080f51a0 2023-12-25T21:11:22.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:22 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:20.608719+0000 mgr.smithi061.jxxeke (mgr.14164) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:22.821 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.821+0000 7f130d528700 1 -- 172.21.15.174:0/1316592361 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1308106de0 con 0x7f13080f51a0 2023-12-25T21:11:22.821 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.822+0000 7f1304ff9700 1 -- 172.21.15.174:0/1316592361 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f12f801cb10 con 0x7f13080f51a0 2023-12-25T21:11:22.822 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:22.822 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:22.823 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.823+0000 7f130d528700 1 -- 172.21.15.174:0/1316592361 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f12f002ffb0 msgr2=0x7f12f0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:22.823 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.823+0000 7f130d528700 1 --2- 172.21.15.174:0/1316592361 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f12f002ffb0 0x7f12f0032470 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f12fc0096a0 tx=0x7f12fc006b40).stop 2023-12-25T21:11:22.823 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.823+0000 7f130d528700 1 -- 172.21.15.174:0/1316592361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13080f51a0 msgr2=0x7f13081053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:22.823 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.823+0000 7f130d528700 1 --2- 172.21.15.174:0/1316592361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13080f51a0 0x7f13081053b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f12f8013510 tx=0x7f12f800b040).stop 2023-12-25T21:11:22.823 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.823+0000 7f130d528700 1 -- 172.21.15.174:0/1316592361 shutdown_connections 2023-12-25T21:11:22.824 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.823+0000 7f130d528700 1 --2- 172.21.15.174:0/1316592361 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f12f002ffb0 0x7f12f0032470 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:22.824 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.823+0000 7f130d528700 1 --2- 172.21.15.174:0/1316592361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13080f51a0 0x7f13081053b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:22.824 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.823+0000 7f130d528700 1 -- 172.21.15.174:0/1316592361 >> 172.21.15.174:0/1316592361 conn(0x7f13080f09a0 msgr2=0x7f13080f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:22.824 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.823+0000 7f130d528700 1 -- 172.21.15.174:0/1316592361 shutdown_connections 2023-12-25T21:11:22.824 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:22.823+0000 7f130d528700 1 -- 172.21.15.174:0/1316592361 wait complete. 2023-12-25T21:11:22.824 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:23.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:22.822219+0000 mon.smithi061 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.174:0/1316592361' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:24.210 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:24.210 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:24.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:24 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:22.609082+0000 mgr.smithi061.jxxeke (mgr.14164) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:25.360 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.359+0000 7f5c70edb700 1 -- 172.21.15.174:0/893945285 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c6c0f6510 msgr2=0x7f5c6c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:25.360 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.359+0000 7f5c70edb700 1 --2- 172.21.15.174:0/893945285 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c6c0f6510 0x7f5c6c0f6930 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f5c60009fb0 tx=0x7f5c60009b30).stop 2023-12-25T21:11:25.360 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.359+0000 7f5c70edb700 1 -- 172.21.15.174:0/893945285 shutdown_connections 2023-12-25T21:11:25.361 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.359+0000 7f5c70edb700 1 --2- 172.21.15.174:0/893945285 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c6c0f6510 0x7f5c6c0f6930 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:25.361 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.359+0000 7f5c70edb700 1 -- 172.21.15.174:0/893945285 >> 172.21.15.174:0/893945285 conn(0x7f5c6c0f1c70 msgr2=0x7f5c6c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:25.361 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.359+0000 7f5c70edb700 1 -- 172.21.15.174:0/893945285 shutdown_connections 2023-12-25T21:11:25.361 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.359+0000 7f5c70edb700 1 -- 172.21.15.174:0/893945285 wait complete. 2023-12-25T21:11:25.361 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.360+0000 7f5c70edb700 1 Processor -- start 2023-12-25T21:11:25.361 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.360+0000 7f5c70edb700 1 -- start start 2023-12-25T21:11:25.361 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.360+0000 7f5c70edb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c6c0f6510 0x7f5c6c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:25.361 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.360+0000 7f5c70edb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c106c20 con 0x7f5c6c0f6510 2023-12-25T21:11:25.361 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.360+0000 7f5c6a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c6c0f6510 0x7f5c6c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:25.361 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.360+0000 7f5c6a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c6c0f6510 0x7f5c6c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:46672/0 (socket says 172.21.15.174:46672) 2023-12-25T21:11:25.361 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.360+0000 7f5c6a59c700 1 -- 172.21.15.174:0/1734991886 learned_addr learned my addr 172.21.15.174:0/1734991886 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:25.361 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.360+0000 7f5c6a59c700 1 -- 172.21.15.174:0/1734991886 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c60005040 con 0x7f5c6c0f6510 2023-12-25T21:11:25.361 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.360+0000 7f5c6a59c700 1 --2- 172.21.15.174:0/1734991886 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c6c0f6510 0x7f5c6c106730 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f5c60013510 tx=0x7f5c6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:25.362 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.361+0000 7f5c5affd700 1 -- 172.21.15.174:0/1734991886 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c60009320 con 0x7f5c6c0f6510 2023-12-25T21:11:25.362 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.361+0000 7f5c70edb700 1 -- 172.21.15.174:0/1734991886 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c6c106e20 con 0x7f5c6c0f6510 2023-12-25T21:11:25.362 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.361+0000 7f5c5affd700 1 -- 172.21.15.174:0/1734991886 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c60013f00 con 0x7f5c6c0f6510 2023-12-25T21:11:25.362 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.361+0000 7f5c5affd700 1 -- 172.21.15.174:0/1734991886 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c60014d90 con 0x7f5c6c0f6510 2023-12-25T21:11:25.362 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.361+0000 7f5c70edb700 1 -- 172.21.15.174:0/1734991886 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c6c1072b0 con 0x7f5c6c0f6510 2023-12-25T21:11:25.363 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.361+0000 7f5c5affd700 1 -- 172.21.15.174:0/1734991886 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c600095f0 con 0x7f5c6c0f6510 2023-12-25T21:11:25.363 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.361+0000 7f5c70edb700 1 -- 172.21.15.174:0/1734991886 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c6c045ad0 con 0x7f5c6c0f6510 2023-12-25T21:11:25.363 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.362+0000 7f5c5affd700 1 --2- 172.21.15.174:0/1734991886 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5c5402ff60 0x7f5c54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:25.363 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.362+0000 7f5c5affd700 1 -- 172.21.15.174:0/1734991886 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c60040fe0 con 0x7f5c6c0f6510 2023-12-25T21:11:25.363 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.362+0000 7f5c69d9b700 1 --2- 172.21.15.174:0/1734991886 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5c5402ff60 0x7f5c54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:25.363 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.362+0000 7f5c69d9b700 1 --2- 172.21.15.174:0/1734991886 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5c5402ff60 0x7f5c54032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f5c5c0096a0 tx=0x7f5c5c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:25.366 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.366+0000 7f5c5affd700 1 -- 172.21.15.174:0/1734991886 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c6001d7f0 con 0x7f5c6c0f6510 2023-12-25T21:11:25.572 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.572+0000 7f5c70edb700 1 -- 172.21.15.174:0/1734991886 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c6c0fbf60 con 0x7f5c6c0f6510 2023-12-25T21:11:25.572 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.572+0000 7f5c5affd700 1 -- 172.21.15.174:0/1734991886 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5c60010230 con 0x7f5c6c0f6510 2023-12-25T21:11:25.572 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:25.572 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:25.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.574+0000 7f5c70edb700 1 -- 172.21.15.174:0/1734991886 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5c5402ff60 msgr2=0x7f5c54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:25.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.574+0000 7f5c70edb700 1 --2- 172.21.15.174:0/1734991886 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5c5402ff60 0x7f5c54032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f5c5c0096a0 tx=0x7f5c5c006b40).stop 2023-12-25T21:11:25.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.574+0000 7f5c70edb700 1 -- 172.21.15.174:0/1734991886 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c6c0f6510 msgr2=0x7f5c6c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:25.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.574+0000 7f5c70edb700 1 --2- 172.21.15.174:0/1734991886 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c6c0f6510 0x7f5c6c106730 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f5c60013510 tx=0x7f5c6000b040).stop 2023-12-25T21:11:25.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.574+0000 7f5c70edb700 1 -- 172.21.15.174:0/1734991886 shutdown_connections 2023-12-25T21:11:25.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.574+0000 7f5c70edb700 1 --2- 172.21.15.174:0/1734991886 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5c5402ff60 0x7f5c54032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:25.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.574+0000 7f5c70edb700 1 --2- 172.21.15.174:0/1734991886 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c6c0f6510 0x7f5c6c106730 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:25.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.574+0000 7f5c70edb700 1 -- 172.21.15.174:0/1734991886 >> 172.21.15.174:0/1734991886 conn(0x7f5c6c0f1c70 msgr2=0x7f5c6c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:25.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.574+0000 7f5c70edb700 1 -- 172.21.15.174:0/1734991886 shutdown_connections 2023-12-25T21:11:25.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:25.574+0000 7f5c70edb700 1 -- 172.21.15.174:0/1734991886 wait complete. 2023-12-25T21:11:25.575 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:26.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:26 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:24.609389+0000 mgr.smithi061.jxxeke (mgr.14164) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:26.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:26 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:25.572877+0000 mon.smithi061 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.174:0/1734991886' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:26.973 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:26.973 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:28.115 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.113+0000 7efeb49f6700 1 -- 172.21.15.174:0/3295243160 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb00601e0 msgr2=0x7efeb00f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:28.115 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.113+0000 7efeb49f6700 1 --2- 172.21.15.174:0/3295243160 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb00601e0 0x7efeb00f8f20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7efea0009fb0 tx=0x7efea0009b30).stop 2023-12-25T21:11:28.115 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.113+0000 7efeb49f6700 1 -- 172.21.15.174:0/3295243160 shutdown_connections 2023-12-25T21:11:28.115 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.113+0000 7efeb49f6700 1 --2- 172.21.15.174:0/3295243160 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb00601e0 0x7efeb00f8f20 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:28.115 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.113+0000 7efeb49f6700 1 -- 172.21.15.174:0/3295243160 >> 172.21.15.174:0/3295243160 conn(0x7efeb00f09a0 msgr2=0x7efeb00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:28.115 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.113+0000 7efeb49f6700 1 -- 172.21.15.174:0/3295243160 shutdown_connections 2023-12-25T21:11:28.115 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.113+0000 7efeb49f6700 1 -- 172.21.15.174:0/3295243160 wait complete. 2023-12-25T21:11:28.116 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.114+0000 7efeb49f6700 1 Processor -- start 2023-12-25T21:11:28.116 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.114+0000 7efeb49f6700 1 -- start start 2023-12-25T21:11:28.116 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.114+0000 7efeb49f6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb00601e0 0x7efeb0106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:28.116 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.114+0000 7efeb49f6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb01069f0 con 0x7efeb00601e0 2023-12-25T21:11:28.116 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.115+0000 7efeaed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb00601e0 0x7efeb0106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:28.116 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.115+0000 7efeaed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb00601e0 0x7efeb0106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:40444/0 (socket says 172.21.15.174:40444) 2023-12-25T21:11:28.116 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.115+0000 7efeaed9d700 1 -- 172.21.15.174:0/619977669 learned_addr learned my addr 172.21.15.174:0/619977669 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:28.116 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.115+0000 7efeaed9d700 1 -- 172.21.15.174:0/619977669 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea0005040 con 0x7efeb00601e0 2023-12-25T21:11:28.116 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.115+0000 7efeaed9d700 1 --2- 172.21.15.174:0/619977669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb00601e0 0x7efeb0106500 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7efea0013510 tx=0x7efea000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:28.116 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.115+0000 7efe977fe700 1 -- 172.21.15.174:0/619977669 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efea0009320 con 0x7efeb00601e0 2023-12-25T21:11:28.116 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.115+0000 7efeb49f6700 1 -- 172.21.15.174:0/619977669 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb0106bf0 con 0x7efeb00601e0 2023-12-25T21:11:28.116 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.115+0000 7efe977fe700 1 -- 172.21.15.174:0/619977669 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efea0013f00 con 0x7efeb00601e0 2023-12-25T21:11:28.116 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.115+0000 7efe977fe700 1 -- 172.21.15.174:0/619977669 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efea0014d90 con 0x7efeb00601e0 2023-12-25T21:11:28.116 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.115+0000 7efeb49f6700 1 -- 172.21.15.174:0/619977669 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb0107080 con 0x7efeb00601e0 2023-12-25T21:11:28.117 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.116+0000 7efe977fe700 1 -- 172.21.15.174:0/619977669 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efea0014420 con 0x7efeb00601e0 2023-12-25T21:11:28.118 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.116+0000 7efeb49f6700 1 -- 172.21.15.174:0/619977669 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb0045ad0 con 0x7efeb00601e0 2023-12-25T21:11:28.118 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.116+0000 7efe977fe700 1 --2- 172.21.15.174:0/619977669 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efe9802ff60 0x7efe98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:28.118 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.116+0000 7efe977fe700 1 -- 172.21.15.174:0/619977669 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efea00410c0 con 0x7efeb00601e0 2023-12-25T21:11:28.118 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.116+0000 7efeae59c700 1 --2- 172.21.15.174:0/619977669 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efe9802ff60 0x7efe98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:28.118 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.117+0000 7efeae59c700 1 --2- 172.21.15.174:0/619977669 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efe9802ff60 0x7efe98032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7efea40096a0 tx=0x7efea4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:28.120 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.121+0000 7efe977fe700 1 -- 172.21.15.174:0/619977669 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efea00095f0 con 0x7efeb00601e0 2023-12-25T21:11:28.327 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.327+0000 7efeb49f6700 1 -- 172.21.15.174:0/619977669 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efeb0107f30 con 0x7efeb00601e0 2023-12-25T21:11:28.328 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.328+0000 7efe977fe700 1 -- 172.21.15.174:0/619977669 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efea001db70 con 0x7efeb00601e0 2023-12-25T21:11:28.328 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:28.328 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:28.330 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.330+0000 7efeb49f6700 1 -- 172.21.15.174:0/619977669 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efe9802ff60 msgr2=0x7efe98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:28.330 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.330+0000 7efeb49f6700 1 --2- 172.21.15.174:0/619977669 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efe9802ff60 0x7efe98032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7efea40096a0 tx=0x7efea4006b40).stop 2023-12-25T21:11:28.330 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.330+0000 7efeb49f6700 1 -- 172.21.15.174:0/619977669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb00601e0 msgr2=0x7efeb0106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:28.330 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.330+0000 7efeb49f6700 1 --2- 172.21.15.174:0/619977669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb00601e0 0x7efeb0106500 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7efea0013510 tx=0x7efea000b040).stop 2023-12-25T21:11:28.330 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.330+0000 7efeb49f6700 1 -- 172.21.15.174:0/619977669 shutdown_connections 2023-12-25T21:11:28.330 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.330+0000 7efeb49f6700 1 --2- 172.21.15.174:0/619977669 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efe9802ff60 0x7efe98032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:28.330 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.330+0000 7efeb49f6700 1 --2- 172.21.15.174:0/619977669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb00601e0 0x7efeb0106500 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:28.330 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.330+0000 7efeb49f6700 1 -- 172.21.15.174:0/619977669 >> 172.21.15.174:0/619977669 conn(0x7efeb00f09a0 msgr2=0x7efeb00f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:28.330 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.330+0000 7efeb49f6700 1 -- 172.21.15.174:0/619977669 shutdown_connections 2023-12-25T21:11:28.331 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:28.330+0000 7efeb49f6700 1 -- 172.21.15.174:0/619977669 wait complete. 2023-12-25T21:11:28.331 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:28.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:28 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:26.609740+0000 mgr.smithi061.jxxeke (mgr.14164) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:29.644 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:29.645 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:29.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:29 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:28.328666+0000 mon.smithi061 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.174:0/619977669' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:30.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:30 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:28.610140+0000 mgr.smithi061.jxxeke (mgr.14164) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:30.883 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.881+0000 7f8a8621a700 1 -- 172.21.15.174:0/1787041667 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a800f51a0 msgr2=0x7f8a800f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:30.883 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.881+0000 7f8a8621a700 1 --2- 172.21.15.174:0/1787041667 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a800f51a0 0x7f8a800f55c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f8a7c009fb0 tx=0x7f8a7c009b30).stop 2023-12-25T21:11:30.883 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.881+0000 7f8a8621a700 1 -- 172.21.15.174:0/1787041667 shutdown_connections 2023-12-25T21:11:30.884 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.881+0000 7f8a8621a700 1 --2- 172.21.15.174:0/1787041667 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a800f51a0 0x7f8a800f55c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:30.884 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.881+0000 7f8a8621a700 1 -- 172.21.15.174:0/1787041667 >> 172.21.15.174:0/1787041667 conn(0x7f8a800f09a0 msgr2=0x7f8a800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:30.884 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.882+0000 7f8a8621a700 1 -- 172.21.15.174:0/1787041667 shutdown_connections 2023-12-25T21:11:30.884 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.882+0000 7f8a8621a700 1 -- 172.21.15.174:0/1787041667 wait complete. 2023-12-25T21:11:30.884 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.882+0000 7f8a8621a700 1 Processor -- start 2023-12-25T21:11:30.884 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.882+0000 7f8a8621a700 1 -- start start 2023-12-25T21:11:30.884 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.882+0000 7f8a8621a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a800f51a0 0x7f8a801053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:30.884 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.882+0000 7f8a8621a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a801058a0 con 0x7f8a800f51a0 2023-12-25T21:11:30.884 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.883+0000 7f8a85218700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a800f51a0 0x7f8a801053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:30.884 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.883+0000 7f8a85218700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a800f51a0 0x7f8a801053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:40458/0 (socket says 172.21.15.174:40458) 2023-12-25T21:11:30.884 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.883+0000 7f8a85218700 1 -- 172.21.15.174:0/2749309551 learned_addr learned my addr 172.21.15.174:0/2749309551 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:30.884 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.883+0000 7f8a85218700 1 -- 172.21.15.174:0/2749309551 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a7c005040 con 0x7f8a800f51a0 2023-12-25T21:11:30.885 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.883+0000 7f8a85218700 1 --2- 172.21.15.174:0/2749309551 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a800f51a0 0x7f8a801053b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f8a7c009830 tx=0x7f8a7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:30.885 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.883+0000 7f8a75ffb700 1 -- 172.21.15.174:0/2749309551 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a7c009320 con 0x7f8a800f51a0 2023-12-25T21:11:30.885 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.883+0000 7f8a8621a700 1 -- 172.21.15.174:0/2749309551 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a80105aa0 con 0x7f8a800f51a0 2023-12-25T21:11:30.885 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.883+0000 7f8a75ffb700 1 -- 172.21.15.174:0/2749309551 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a7c0136c0 con 0x7f8a800f51a0 2023-12-25T21:11:30.885 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.883+0000 7f8a75ffb700 1 -- 172.21.15.174:0/2749309551 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a7c01ee90 con 0x7f8a800f51a0 2023-12-25T21:11:30.885 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.884+0000 7f8a8621a700 1 -- 172.21.15.174:0/2749309551 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a80105f30 con 0x7f8a800f51a0 2023-12-25T21:11:30.886 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.884+0000 7f8a75ffb700 1 -- 172.21.15.174:0/2749309551 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a7c014420 con 0x7f8a800f51a0 2023-12-25T21:11:30.886 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.884+0000 7f8a75ffb700 1 --2- 172.21.15.174:0/2749309551 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8a6c02ff10 0x7f8a6c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:30.886 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.885+0000 7f8a75ffb700 1 -- 172.21.15.174:0/2749309551 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a7c041020 con 0x7f8a800f51a0 2023-12-25T21:11:30.886 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.885+0000 7f8a8621a700 1 -- 172.21.15.174:0/2749309551 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a80045ad0 con 0x7f8a800f51a0 2023-12-25T21:11:30.886 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.885+0000 7f8a84a17700 1 --2- 172.21.15.174:0/2749309551 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8a6c02ff10 0x7f8a6c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:30.886 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.885+0000 7f8a84a17700 1 --2- 172.21.15.174:0/2749309551 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8a6c02ff10 0x7f8a6c0323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f8a700096a0 tx=0x7f8a70006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:30.889 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:30.890+0000 7f8a75ffb700 1 -- 172.21.15.174:0/2749309551 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a7c01c910 con 0x7f8a800f51a0 2023-12-25T21:11:31.093 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:31.093+0000 7f8a8621a700 1 -- 172.21.15.174:0/2749309551 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a80106c70 con 0x7f8a800f51a0 2023-12-25T21:11:31.093 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:31.093+0000 7f8a75ffb700 1 -- 172.21.15.174:0/2749309551 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8a7c01c320 con 0x7f8a800f51a0 2023-12-25T21:11:31.094 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:31.094 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:31.095 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:31.095+0000 7f8a8621a700 1 -- 172.21.15.174:0/2749309551 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8a6c02ff10 msgr2=0x7f8a6c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:31.095 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:31.095+0000 7f8a8621a700 1 --2- 172.21.15.174:0/2749309551 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8a6c02ff10 0x7f8a6c0323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f8a700096a0 tx=0x7f8a70006b40).stop 2023-12-25T21:11:31.095 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:31.095+0000 7f8a8621a700 1 -- 172.21.15.174:0/2749309551 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a800f51a0 msgr2=0x7f8a801053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:31.095 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:31.095+0000 7f8a8621a700 1 --2- 172.21.15.174:0/2749309551 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a800f51a0 0x7f8a801053b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f8a7c009830 tx=0x7f8a7c00b040).stop 2023-12-25T21:11:31.095 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:31.095+0000 7f8a8621a700 1 -- 172.21.15.174:0/2749309551 shutdown_connections 2023-12-25T21:11:31.095 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:31.095+0000 7f8a8621a700 1 --2- 172.21.15.174:0/2749309551 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8a6c02ff10 0x7f8a6c0323d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:31.095 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:31.095+0000 7f8a8621a700 1 --2- 172.21.15.174:0/2749309551 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a800f51a0 0x7f8a801053b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:31.095 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:31.095+0000 7f8a8621a700 1 -- 172.21.15.174:0/2749309551 >> 172.21.15.174:0/2749309551 conn(0x7f8a800f09a0 msgr2=0x7f8a800f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:31.095 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:31.095+0000 7f8a8621a700 1 -- 172.21.15.174:0/2749309551 shutdown_connections 2023-12-25T21:11:31.095 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:31.095+0000 7f8a8621a700 1 -- 172.21.15.174:0/2749309551 wait complete. 2023-12-25T21:11:31.096 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:31.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:31 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:31.093998+0000 mon.smithi061 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.174:0/2749309551' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:32.623 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:32.624 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:32.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:32 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:30.610501+0000 mgr.smithi061.jxxeke (mgr.14164) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:33.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.817+0000 7f3139968700 1 -- 172.21.15.174:0/1329904549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31340f47c0 msgr2=0x7f31340f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:33.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.817+0000 7f3139968700 1 --2- 172.21.15.174:0/1329904549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31340f47c0 0x7f31340f4be0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f3124009fb0 tx=0x7f3124009b30).stop 2023-12-25T21:11:33.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.817+0000 7f3139968700 1 -- 172.21.15.174:0/1329904549 shutdown_connections 2023-12-25T21:11:33.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.817+0000 7f3139968700 1 --2- 172.21.15.174:0/1329904549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31340f47c0 0x7f31340f4be0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:33.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.817+0000 7f3139968700 1 -- 172.21.15.174:0/1329904549 >> 172.21.15.174:0/1329904549 conn(0x7f31340f02e0 msgr2=0x7f31340f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:33.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.817+0000 7f3139968700 1 -- 172.21.15.174:0/1329904549 shutdown_connections 2023-12-25T21:11:33.818 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.817+0000 7f3139968700 1 -- 172.21.15.174:0/1329904549 wait complete. 2023-12-25T21:11:33.820 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.817+0000 7f3139968700 1 Processor -- start 2023-12-25T21:11:33.820 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.817+0000 7f3139968700 1 -- start start 2023-12-25T21:11:33.820 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.818+0000 7f3139968700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31340f47c0 0x7f3134106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:33.820 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.818+0000 7f3139968700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3134106bf0 con 0x7f31340f47c0 2023-12-25T21:11:33.820 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.818+0000 7f3132ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31340f47c0 0x7f3134106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:33.820 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.818+0000 7f3132ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31340f47c0 0x7f3134106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:40476/0 (socket says 172.21.15.174:40476) 2023-12-25T21:11:33.821 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.818+0000 7f3132ffd700 1 -- 172.21.15.174:0/2032696569 learned_addr learned my addr 172.21.15.174:0/2032696569 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:33.821 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.818+0000 7f3132ffd700 1 -- 172.21.15.174:0/2032696569 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3124005040 con 0x7f31340f47c0 2023-12-25T21:11:33.821 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.818+0000 7f3132ffd700 1 --2- 172.21.15.174:0/2032696569 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31340f47c0 0x7f3134106700 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f3124013510 tx=0x7f312400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:33.821 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.818+0000 7f311b7fe700 1 -- 172.21.15.174:0/2032696569 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3124009320 con 0x7f31340f47c0 2023-12-25T21:11:33.821 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.818+0000 7f3139968700 1 -- 172.21.15.174:0/2032696569 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3134106df0 con 0x7f31340f47c0 2023-12-25T21:11:33.821 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.819+0000 7f311b7fe700 1 -- 172.21.15.174:0/2032696569 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3124013f00 con 0x7f31340f47c0 2023-12-25T21:11:33.821 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.819+0000 7f311b7fe700 1 -- 172.21.15.174:0/2032696569 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3124014d90 con 0x7f31340f47c0 2023-12-25T21:11:33.821 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.819+0000 7f3139968700 1 -- 172.21.15.174:0/2032696569 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3134107280 con 0x7f31340f47c0 2023-12-25T21:11:33.821 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.819+0000 7f311b7fe700 1 -- 172.21.15.174:0/2032696569 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3124014420 con 0x7f31340f47c0 2023-12-25T21:11:33.821 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.819+0000 7f3139968700 1 -- 172.21.15.174:0/2032696569 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3134045ad0 con 0x7f31340f47c0 2023-12-25T21:11:33.821 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.819+0000 7f311b7fe700 1 --2- 172.21.15.174:0/2032696569 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f311c02ff60 0x7f311c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:33.821 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.820+0000 7f311b7fe700 1 -- 172.21.15.174:0/2032696569 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f31240410c0 con 0x7f31340f47c0 2023-12-25T21:11:33.821 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.820+0000 7f31327fc700 1 --2- 172.21.15.174:0/2032696569 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f311c02ff60 0x7f311c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:33.822 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.820+0000 7f31327fc700 1 --2- 172.21.15.174:0/2032696569 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f311c02ff60 0x7f311c032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f31280096a0 tx=0x7f3128006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:33.825 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:33.824+0000 7f311b7fe700 1 -- 172.21.15.174:0/2032696569 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f31240095f0 con 0x7f31340f47c0 2023-12-25T21:11:34.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:34.030+0000 7f3139968700 1 -- 172.21.15.174:0/2032696569 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3134108130 con 0x7f31340f47c0 2023-12-25T21:11:34.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:34.031+0000 7f311b7fe700 1 -- 172.21.15.174:0/2032696569 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f312401db70 con 0x7f31340f47c0 2023-12-25T21:11:34.032 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:34.032 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:34.033 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:34.032+0000 7f3139968700 1 -- 172.21.15.174:0/2032696569 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f311c02ff60 msgr2=0x7f311c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:34.033 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:34.032+0000 7f3139968700 1 --2- 172.21.15.174:0/2032696569 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f311c02ff60 0x7f311c032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f31280096a0 tx=0x7f3128006b40).stop 2023-12-25T21:11:34.033 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:34.032+0000 7f3139968700 1 -- 172.21.15.174:0/2032696569 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31340f47c0 msgr2=0x7f3134106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:34.033 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:34.032+0000 7f3139968700 1 --2- 172.21.15.174:0/2032696569 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31340f47c0 0x7f3134106700 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f3124013510 tx=0x7f312400b040).stop 2023-12-25T21:11:34.033 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:34.032+0000 7f3139968700 1 -- 172.21.15.174:0/2032696569 shutdown_connections 2023-12-25T21:11:34.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:34.032+0000 7f3139968700 1 --2- 172.21.15.174:0/2032696569 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f311c02ff60 0x7f311c032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:34.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:34.032+0000 7f3139968700 1 --2- 172.21.15.174:0/2032696569 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31340f47c0 0x7f3134106700 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:34.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:34.032+0000 7f3139968700 1 -- 172.21.15.174:0/2032696569 >> 172.21.15.174:0/2032696569 conn(0x7f31340f02e0 msgr2=0x7f31340f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:34.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:34.032+0000 7f3139968700 1 -- 172.21.15.174:0/2032696569 shutdown_connections 2023-12-25T21:11:34.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:34.032+0000 7f3139968700 1 -- 172.21.15.174:0/2032696569 wait complete. 2023-12-25T21:11:34.034 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:34.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:34 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:32.610838+0000 mgr.smithi061.jxxeke (mgr.14164) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:34.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:34 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:34.032311+0000 mon.smithi061 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.174:0/2032696569' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:35.444 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:35.445 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:36.639 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.637+0000 7ff8c564b700 1 -- 172.21.15.174:0/478585764 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8c00f5410 msgr2=0x7ff8c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:36.639 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.637+0000 7ff8c564b700 1 --2- 172.21.15.174:0/478585764 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8c00f5410 0x7ff8c00f5830 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7ff8b4009fb0 tx=0x7ff8b4009b30).stop 2023-12-25T21:11:36.639 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.637+0000 7ff8c564b700 1 -- 172.21.15.174:0/478585764 shutdown_connections 2023-12-25T21:11:36.639 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.637+0000 7ff8c564b700 1 --2- 172.21.15.174:0/478585764 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8c00f5410 0x7ff8c00f5830 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:36.639 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.637+0000 7ff8c564b700 1 -- 172.21.15.174:0/478585764 >> 172.21.15.174:0/478585764 conn(0x7ff8c00f0b70 msgr2=0x7ff8c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:36.639 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.637+0000 7ff8c564b700 1 -- 172.21.15.174:0/478585764 shutdown_connections 2023-12-25T21:11:36.640 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.637+0000 7ff8c564b700 1 -- 172.21.15.174:0/478585764 wait complete. 2023-12-25T21:11:36.640 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.638+0000 7ff8c564b700 1 Processor -- start 2023-12-25T21:11:36.640 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.638+0000 7ff8c564b700 1 -- start start 2023-12-25T21:11:36.640 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.638+0000 7ff8c564b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8c00f5410 0x7ff8c0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:36.640 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.638+0000 7ff8c564b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c0106c20 con 0x7ff8c00f5410 2023-12-25T21:11:36.640 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.638+0000 7ff8beffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8c00f5410 0x7ff8c0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:36.640 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.638+0000 7ff8beffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8c00f5410 0x7ff8c0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:55922/0 (socket says 172.21.15.174:55922) 2023-12-25T21:11:36.640 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.638+0000 7ff8beffd700 1 -- 172.21.15.174:0/4067259924 learned_addr learned my addr 172.21.15.174:0/4067259924 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:36.641 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.639+0000 7ff8beffd700 1 -- 172.21.15.174:0/4067259924 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b4005040 con 0x7ff8c00f5410 2023-12-25T21:11:36.642 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.639+0000 7ff8beffd700 1 --2- 172.21.15.174:0/4067259924 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8c00f5410 0x7ff8c0106730 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7ff8b4013510 tx=0x7ff8b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:36.642 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.639+0000 7ff8a77fe700 1 -- 172.21.15.174:0/4067259924 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8b4009320 con 0x7ff8c00f5410 2023-12-25T21:11:36.642 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.639+0000 7ff8c564b700 1 -- 172.21.15.174:0/4067259924 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c0106e20 con 0x7ff8c00f5410 2023-12-25T21:11:36.642 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.639+0000 7ff8a77fe700 1 -- 172.21.15.174:0/4067259924 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8b4013f00 con 0x7ff8c00f5410 2023-12-25T21:11:36.642 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.639+0000 7ff8a77fe700 1 -- 172.21.15.174:0/4067259924 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8b4014d90 con 0x7ff8c00f5410 2023-12-25T21:11:36.642 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.639+0000 7ff8c564b700 1 -- 172.21.15.174:0/4067259924 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c01072b0 con 0x7ff8c00f5410 2023-12-25T21:11:36.642 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.640+0000 7ff8a77fe700 1 -- 172.21.15.174:0/4067259924 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8b40095f0 con 0x7ff8c00f5410 2023-12-25T21:11:36.642 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.640+0000 7ff8c564b700 1 -- 172.21.15.174:0/4067259924 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c0045ad0 con 0x7ff8c00f5410 2023-12-25T21:11:36.642 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.640+0000 7ff8a77fe700 1 --2- 172.21.15.174:0/4067259924 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff8a802ff60 0x7ff8a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:36.642 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.640+0000 7ff8a77fe700 1 -- 172.21.15.174:0/4067259924 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff8b4040fd0 con 0x7ff8c00f5410 2023-12-25T21:11:36.642 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.640+0000 7ff8be7fc700 1 --2- 172.21.15.174:0/4067259924 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff8a802ff60 0x7ff8a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:36.642 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.641+0000 7ff8be7fc700 1 --2- 172.21.15.174:0/4067259924 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff8a802ff60 0x7ff8a8032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7ff8b00096a0 tx=0x7ff8b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:36.645 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.645+0000 7ff8a77fe700 1 -- 172.21.15.174:0/4067259924 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8b4014420 con 0x7ff8c00f5410 2023-12-25T21:11:36.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:36 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:34.611170+0000 mgr.smithi061.jxxeke (mgr.14164) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:36.851 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.850+0000 7ff8c564b700 1 -- 172.21.15.174:0/4067259924 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8c00fbf90 con 0x7ff8c00f5410 2023-12-25T21:11:36.851 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.850+0000 7ff8a77fe700 1 -- 172.21.15.174:0/4067259924 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff8b401d7f0 con 0x7ff8c00f5410 2023-12-25T21:11:36.852 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:36.852 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:36.853 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.852+0000 7ff8c564b700 1 -- 172.21.15.174:0/4067259924 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff8a802ff60 msgr2=0x7ff8a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:36.853 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.852+0000 7ff8c564b700 1 --2- 172.21.15.174:0/4067259924 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff8a802ff60 0x7ff8a8032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7ff8b00096a0 tx=0x7ff8b0006b40).stop 2023-12-25T21:11:36.853 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.852+0000 7ff8c564b700 1 -- 172.21.15.174:0/4067259924 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8c00f5410 msgr2=0x7ff8c0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:36.853 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.852+0000 7ff8c564b700 1 --2- 172.21.15.174:0/4067259924 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8c00f5410 0x7ff8c0106730 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7ff8b4013510 tx=0x7ff8b400b040).stop 2023-12-25T21:11:36.853 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.852+0000 7ff8c564b700 1 -- 172.21.15.174:0/4067259924 shutdown_connections 2023-12-25T21:11:36.853 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.852+0000 7ff8c564b700 1 --2- 172.21.15.174:0/4067259924 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff8a802ff60 0x7ff8a8032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:36.853 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.852+0000 7ff8c564b700 1 --2- 172.21.15.174:0/4067259924 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8c00f5410 0x7ff8c0106730 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:36.853 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.852+0000 7ff8c564b700 1 -- 172.21.15.174:0/4067259924 >> 172.21.15.174:0/4067259924 conn(0x7ff8c00f0b70 msgr2=0x7ff8c00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:36.853 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.852+0000 7ff8c564b700 1 -- 172.21.15.174:0/4067259924 shutdown_connections 2023-12-25T21:11:36.854 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:36.852+0000 7ff8c564b700 1 -- 172.21.15.174:0/4067259924 wait complete. 2023-12-25T21:11:36.854 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:37.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:37 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:36.851820+0000 mon.smithi061 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.174:0/4067259924' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:38.247 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:38.248 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:38.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:38 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:36.611560+0000 mgr.smithi061.jxxeke (mgr.14164) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:39.411 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.409+0000 7fd715fed700 1 -- 172.21.15.174:0/621257680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7100f51a0 msgr2=0x7fd7100f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:39.411 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.409+0000 7fd715fed700 1 --2- 172.21.15.174:0/621257680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7100f51a0 0x7fd7100f55c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fd704009fb0 tx=0x7fd704009b30).stop 2023-12-25T21:11:39.411 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.410+0000 7fd715fed700 1 -- 172.21.15.174:0/621257680 shutdown_connections 2023-12-25T21:11:39.411 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.410+0000 7fd715fed700 1 --2- 172.21.15.174:0/621257680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7100f51a0 0x7fd7100f55c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:39.411 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.410+0000 7fd715fed700 1 -- 172.21.15.174:0/621257680 >> 172.21.15.174:0/621257680 conn(0x7fd7100f09a0 msgr2=0x7fd7100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:39.411 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.410+0000 7fd715fed700 1 -- 172.21.15.174:0/621257680 shutdown_connections 2023-12-25T21:11:39.411 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.410+0000 7fd715fed700 1 -- 172.21.15.174:0/621257680 wait complete. 2023-12-25T21:11:39.411 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.410+0000 7fd715fed700 1 Processor -- start 2023-12-25T21:11:39.411 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.410+0000 7fd715fed700 1 -- start start 2023-12-25T21:11:39.413 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.410+0000 7fd715fed700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7100f51a0 0x7fd7101064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:39.413 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.410+0000 7fd715fed700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7101069b0 con 0x7fd7100f51a0 2023-12-25T21:11:39.413 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.411+0000 7fd714feb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7100f51a0 0x7fd7101064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:39.413 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.411+0000 7fd714feb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7100f51a0 0x7fd7101064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:55938/0 (socket says 172.21.15.174:55938) 2023-12-25T21:11:39.413 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.411+0000 7fd714feb700 1 -- 172.21.15.174:0/3521163796 learned_addr learned my addr 172.21.15.174:0/3521163796 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:39.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.411+0000 7fd714feb700 1 -- 172.21.15.174:0/3521163796 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd704005040 con 0x7fd7100f51a0 2023-12-25T21:11:39.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.411+0000 7fd714feb700 1 --2- 172.21.15.174:0/3521163796 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7100f51a0 0x7fd7101064c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fd704013510 tx=0x7fd70400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:39.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.411+0000 7fd70d7fa700 1 -- 172.21.15.174:0/3521163796 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd704009320 con 0x7fd7100f51a0 2023-12-25T21:11:39.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.411+0000 7fd70d7fa700 1 -- 172.21.15.174:0/3521163796 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd704013f00 con 0x7fd7100f51a0 2023-12-25T21:11:39.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.411+0000 7fd715fed700 1 -- 172.21.15.174:0/3521163796 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd710106bb0 con 0x7fd7100f51a0 2023-12-25T21:11:39.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.411+0000 7fd70d7fa700 1 -- 172.21.15.174:0/3521163796 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd704014dc0 con 0x7fd7100f51a0 2023-12-25T21:11:39.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.411+0000 7fd715fed700 1 -- 172.21.15.174:0/3521163796 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd710107040 con 0x7fd7100f51a0 2023-12-25T21:11:39.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.412+0000 7fd70d7fa700 1 -- 172.21.15.174:0/3521163796 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd7040095f0 con 0x7fd7100f51a0 2023-12-25T21:11:39.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.412+0000 7fd715fed700 1 -- 172.21.15.174:0/3521163796 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd710045ad0 con 0x7fd7100f51a0 2023-12-25T21:11:39.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.413+0000 7fd70d7fa700 1 --2- 172.21.15.174:0/3521163796 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd70002ffb0 0x7fd700032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:39.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.413+0000 7fd70d7fa700 1 -- 172.21.15.174:0/3521163796 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd704040fe0 con 0x7fd7100f51a0 2023-12-25T21:11:39.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.413+0000 7fd70ffff700 1 --2- 172.21.15.174:0/3521163796 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd70002ffb0 0x7fd700032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:39.415 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.413+0000 7fd70ffff700 1 --2- 172.21.15.174:0/3521163796 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd70002ffb0 0x7fd700032470 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fd6fc0096a0 tx=0x7fd6fc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:39.418 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.417+0000 7fd70d7fa700 1 -- 172.21.15.174:0/3521163796 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd70406d2b0 con 0x7fd7100f51a0 2023-12-25T21:11:39.626 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.625+0000 7fd715fed700 1 -- 172.21.15.174:0/3521163796 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd710107ef0 con 0x7fd7100f51a0 2023-12-25T21:11:39.626 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.626+0000 7fd70d7fa700 1 -- 172.21.15.174:0/3521163796 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd704014420 con 0x7fd7100f51a0 2023-12-25T21:11:39.627 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:39.627 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:39.628 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.627+0000 7fd715fed700 1 -- 172.21.15.174:0/3521163796 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd70002ffb0 msgr2=0x7fd700032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:39.628 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.627+0000 7fd715fed700 1 --2- 172.21.15.174:0/3521163796 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd70002ffb0 0x7fd700032470 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fd6fc0096a0 tx=0x7fd6fc006b40).stop 2023-12-25T21:11:39.629 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.627+0000 7fd715fed700 1 -- 172.21.15.174:0/3521163796 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7100f51a0 msgr2=0x7fd7101064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:39.629 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.627+0000 7fd715fed700 1 --2- 172.21.15.174:0/3521163796 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7100f51a0 0x7fd7101064c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fd704013510 tx=0x7fd70400b040).stop 2023-12-25T21:11:39.629 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.627+0000 7fd715fed700 1 -- 172.21.15.174:0/3521163796 shutdown_connections 2023-12-25T21:11:39.629 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.627+0000 7fd715fed700 1 --2- 172.21.15.174:0/3521163796 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd70002ffb0 0x7fd700032470 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:39.629 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.628+0000 7fd715fed700 1 --2- 172.21.15.174:0/3521163796 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7100f51a0 0x7fd7101064c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:39.629 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.628+0000 7fd715fed700 1 -- 172.21.15.174:0/3521163796 >> 172.21.15.174:0/3521163796 conn(0x7fd7100f09a0 msgr2=0x7fd7100f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:39.629 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.628+0000 7fd715fed700 1 -- 172.21.15.174:0/3521163796 shutdown_connections 2023-12-25T21:11:39.629 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:39.628+0000 7fd715fed700 1 -- 172.21.15.174:0/3521163796 wait complete. 2023-12-25T21:11:39.629 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:40.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:40 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:38.611909+0000 mgr.smithi061.jxxeke (mgr.14164) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:40.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:40 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:39.627189+0000 mon.smithi061 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.174:0/3521163796' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:41.026 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:41.027 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:42.259 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.254+0000 7fdb658b7700 1 -- 172.21.15.174:0/4115330172 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb600f6510 msgr2=0x7fdb600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:42.259 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.254+0000 7fdb658b7700 1 --2- 172.21.15.174:0/4115330172 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb600f6510 0x7fdb600f6930 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fdb50009fb0 tx=0x7fdb50009b30).stop 2023-12-25T21:11:42.259 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.254+0000 7fdb658b7700 1 -- 172.21.15.174:0/4115330172 shutdown_connections 2023-12-25T21:11:42.259 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.254+0000 7fdb658b7700 1 --2- 172.21.15.174:0/4115330172 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb600f6510 0x7fdb600f6930 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:42.259 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.254+0000 7fdb658b7700 1 -- 172.21.15.174:0/4115330172 >> 172.21.15.174:0/4115330172 conn(0x7fdb600f1c70 msgr2=0x7fdb600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:42.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.254+0000 7fdb658b7700 1 -- 172.21.15.174:0/4115330172 shutdown_connections 2023-12-25T21:11:42.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.255+0000 7fdb658b7700 1 -- 172.21.15.174:0/4115330172 wait complete. 2023-12-25T21:11:42.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.255+0000 7fdb658b7700 1 Processor -- start 2023-12-25T21:11:42.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.255+0000 7fdb658b7700 1 -- start start 2023-12-25T21:11:42.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.255+0000 7fdb658b7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb600f6510 0x7fdb60106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:42.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.255+0000 7fdb658b7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb60106c20 con 0x7fdb600f6510 2023-12-25T21:11:42.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.256+0000 7fdb5effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb600f6510 0x7fdb60106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:42.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.256+0000 7fdb5effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb600f6510 0x7fdb60106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:55950/0 (socket says 172.21.15.174:55950) 2023-12-25T21:11:42.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.256+0000 7fdb5effd700 1 -- 172.21.15.174:0/2665763718 learned_addr learned my addr 172.21.15.174:0/2665763718 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:42.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.256+0000 7fdb5effd700 1 -- 172.21.15.174:0/2665763718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb50005040 con 0x7fdb600f6510 2023-12-25T21:11:42.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.256+0000 7fdb5effd700 1 --2- 172.21.15.174:0/2665763718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb600f6510 0x7fdb60106730 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fdb50013510 tx=0x7fdb5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:42.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.256+0000 7fdb477fe700 1 -- 172.21.15.174:0/2665763718 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb50009320 con 0x7fdb600f6510 2023-12-25T21:11:42.261 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.256+0000 7fdb658b7700 1 -- 172.21.15.174:0/2665763718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb60106e20 con 0x7fdb600f6510 2023-12-25T21:11:42.261 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.256+0000 7fdb477fe700 1 -- 172.21.15.174:0/2665763718 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb50013f00 con 0x7fdb600f6510 2023-12-25T21:11:42.261 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.256+0000 7fdb477fe700 1 -- 172.21.15.174:0/2665763718 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb50014d90 con 0x7fdb600f6510 2023-12-25T21:11:42.261 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.256+0000 7fdb658b7700 1 -- 172.21.15.174:0/2665763718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb601072b0 con 0x7fdb600f6510 2023-12-25T21:11:42.261 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.257+0000 7fdb477fe700 1 -- 172.21.15.174:0/2665763718 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdb50014420 con 0x7fdb600f6510 2023-12-25T21:11:42.261 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.257+0000 7fdb658b7700 1 -- 172.21.15.174:0/2665763718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb60045ad0 con 0x7fdb600f6510 2023-12-25T21:11:42.261 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.257+0000 7fdb477fe700 1 --2- 172.21.15.174:0/2665763718 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdb4802ff60 0x7fdb48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:42.261 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.257+0000 7fdb477fe700 1 -- 172.21.15.174:0/2665763718 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdb50040fb0 con 0x7fdb600f6510 2023-12-25T21:11:42.261 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.257+0000 7fdb5e7fc700 1 --2- 172.21.15.174:0/2665763718 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdb4802ff60 0x7fdb48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:42.261 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.258+0000 7fdb5e7fc700 1 --2- 172.21.15.174:0/2665763718 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdb4802ff60 0x7fdb48032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fdb540096a0 tx=0x7fdb54006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:42.262 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.262+0000 7fdb477fe700 1 -- 172.21.15.174:0/2665763718 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb500095f0 con 0x7fdb600f6510 2023-12-25T21:11:42.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.471+0000 7fdb658b7700 1 -- 172.21.15.174:0/2665763718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb600fbf60 con 0x7fdb600f6510 2023-12-25T21:11:42.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.471+0000 7fdb477fe700 1 -- 172.21.15.174:0/2665763718 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdb5001db70 con 0x7fdb600f6510 2023-12-25T21:11:42.473 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:42.473 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:42.474 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.473+0000 7fdb658b7700 1 -- 172.21.15.174:0/2665763718 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdb4802ff60 msgr2=0x7fdb48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:42.474 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.473+0000 7fdb658b7700 1 --2- 172.21.15.174:0/2665763718 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdb4802ff60 0x7fdb48032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fdb540096a0 tx=0x7fdb54006b40).stop 2023-12-25T21:11:42.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.473+0000 7fdb658b7700 1 -- 172.21.15.174:0/2665763718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb600f6510 msgr2=0x7fdb60106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:42.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.473+0000 7fdb658b7700 1 --2- 172.21.15.174:0/2665763718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb600f6510 0x7fdb60106730 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fdb50013510 tx=0x7fdb5000b040).stop 2023-12-25T21:11:42.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.473+0000 7fdb658b7700 1 -- 172.21.15.174:0/2665763718 shutdown_connections 2023-12-25T21:11:42.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.473+0000 7fdb658b7700 1 --2- 172.21.15.174:0/2665763718 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fdb4802ff60 0x7fdb48032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:42.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.473+0000 7fdb658b7700 1 --2- 172.21.15.174:0/2665763718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb600f6510 0x7fdb60106730 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:42.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.473+0000 7fdb658b7700 1 -- 172.21.15.174:0/2665763718 >> 172.21.15.174:0/2665763718 conn(0x7fdb600f1c70 msgr2=0x7fdb600f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:42.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.473+0000 7fdb658b7700 1 -- 172.21.15.174:0/2665763718 shutdown_connections 2023-12-25T21:11:42.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:42.473+0000 7fdb658b7700 1 -- 172.21.15.174:0/2665763718 wait complete. 2023-12-25T21:11:42.476 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:42.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:42 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:40.612305+0000 mgr.smithi061.jxxeke (mgr.14164) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:43.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:43 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:42.472737+0000 mon.smithi061 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.174:0/2665763718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:43.862 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:43.862 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:44.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:44 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:42.612656+0000 mgr.smithi061.jxxeke (mgr.14164) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:44.990 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.989+0000 7f2d2eee2700 1 -- 172.21.15.174:0/2926668234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d280f51a0 msgr2=0x7f2d280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:44.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.989+0000 7f2d2eee2700 1 --2- 172.21.15.174:0/2926668234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d280f51a0 0x7f2d280f55c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f2d18009fb0 tx=0x7f2d18009b30).stop 2023-12-25T21:11:44.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.989+0000 7f2d2eee2700 1 -- 172.21.15.174:0/2926668234 shutdown_connections 2023-12-25T21:11:44.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.989+0000 7f2d2eee2700 1 --2- 172.21.15.174:0/2926668234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d280f51a0 0x7f2d280f55c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:44.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.989+0000 7f2d2eee2700 1 -- 172.21.15.174:0/2926668234 >> 172.21.15.174:0/2926668234 conn(0x7f2d280f09a0 msgr2=0x7f2d280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:44.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.989+0000 7f2d2eee2700 1 -- 172.21.15.174:0/2926668234 shutdown_connections 2023-12-25T21:11:44.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.989+0000 7f2d2eee2700 1 -- 172.21.15.174:0/2926668234 wait complete. 2023-12-25T21:11:44.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.989+0000 7f2d2eee2700 1 Processor -- start 2023-12-25T21:11:44.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.989+0000 7f2d2eee2700 1 -- start start 2023-12-25T21:11:44.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.989+0000 7f2d2eee2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d280f51a0 0x7f2d281064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:44.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.989+0000 7f2d2eee2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d281069d0 con 0x7f2d280f51a0 2023-12-25T21:11:44.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.990+0000 7f2d2dee0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d280f51a0 0x7f2d281064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:44.992 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.990+0000 7f2d2dee0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d280f51a0 0x7f2d281064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:55960/0 (socket says 172.21.15.174:55960) 2023-12-25T21:11:44.992 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.990+0000 7f2d2dee0700 1 -- 172.21.15.174:0/3383272583 learned_addr learned my addr 172.21.15.174:0/3383272583 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:44.992 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.990+0000 7f2d2dee0700 1 -- 172.21.15.174:0/3383272583 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d18005040 con 0x7f2d280f51a0 2023-12-25T21:11:44.992 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.990+0000 7f2d2dee0700 1 --2- 172.21.15.174:0/3383272583 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d280f51a0 0x7f2d281064e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f2d18013510 tx=0x7f2d1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:44.993 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.990+0000 7f2d1e7fc700 1 -- 172.21.15.174:0/3383272583 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d18009320 con 0x7f2d280f51a0 2023-12-25T21:11:44.993 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.990+0000 7f2d2eee2700 1 -- 172.21.15.174:0/3383272583 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d28106bd0 con 0x7f2d280f51a0 2023-12-25T21:11:44.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.990+0000 7f2d1e7fc700 1 -- 172.21.15.174:0/3383272583 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d18013f00 con 0x7f2d280f51a0 2023-12-25T21:11:44.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.990+0000 7f2d1e7fc700 1 -- 172.21.15.174:0/3383272583 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d18014dc0 con 0x7f2d280f51a0 2023-12-25T21:11:44.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.990+0000 7f2d2eee2700 1 -- 172.21.15.174:0/3383272583 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d28107060 con 0x7f2d280f51a0 2023-12-25T21:11:44.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.991+0000 7f2d1e7fc700 1 -- 172.21.15.174:0/3383272583 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d180095f0 con 0x7f2d280f51a0 2023-12-25T21:11:44.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.991+0000 7f2d2eee2700 1 -- 172.21.15.174:0/3383272583 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d28045ad0 con 0x7f2d280f51a0 2023-12-25T21:11:44.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.991+0000 7f2d1e7fc700 1 --2- 172.21.15.174:0/3383272583 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2d1402ff60 0x7f2d14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:44.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.991+0000 7f2d1e7fc700 1 -- 172.21.15.174:0/3383272583 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d18041100 con 0x7f2d280f51a0 2023-12-25T21:11:44.995 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.992+0000 7f2d2d6df700 1 --2- 172.21.15.174:0/3383272583 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2d1402ff60 0x7f2d14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:44.995 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.992+0000 7f2d2d6df700 1 --2- 172.21.15.174:0/3383272583 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2d1402ff60 0x7f2d14032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f2d240096a0 tx=0x7f2d24006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:44.997 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:44.996+0000 7f2d1e7fc700 1 -- 172.21.15.174:0/3383272583 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d18014420 con 0x7f2d280f51a0 2023-12-25T21:11:45.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:45.201+0000 7f2d2eee2700 1 -- 172.21.15.174:0/3383272583 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d28107f10 con 0x7f2d280f51a0 2023-12-25T21:11:45.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:45.201+0000 7f2d1e7fc700 1 -- 172.21.15.174:0/3383272583 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d18010230 con 0x7f2d280f51a0 2023-12-25T21:11:45.203 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:45.203 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:45.204 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:45.203+0000 7f2d2eee2700 1 -- 172.21.15.174:0/3383272583 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2d1402ff60 msgr2=0x7f2d14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:45.204 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:45.203+0000 7f2d2eee2700 1 --2- 172.21.15.174:0/3383272583 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2d1402ff60 0x7f2d14032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f2d240096a0 tx=0x7f2d24006b40).stop 2023-12-25T21:11:45.204 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:45.203+0000 7f2d2eee2700 1 -- 172.21.15.174:0/3383272583 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d280f51a0 msgr2=0x7f2d281064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:45.205 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:45.203+0000 7f2d2eee2700 1 --2- 172.21.15.174:0/3383272583 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d280f51a0 0x7f2d281064e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f2d18013510 tx=0x7f2d1800b040).stop 2023-12-25T21:11:45.205 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:45.203+0000 7f2d2eee2700 1 -- 172.21.15.174:0/3383272583 shutdown_connections 2023-12-25T21:11:45.205 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:45.203+0000 7f2d2eee2700 1 --2- 172.21.15.174:0/3383272583 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2d1402ff60 0x7f2d14032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:45.205 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:45.203+0000 7f2d2eee2700 1 --2- 172.21.15.174:0/3383272583 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d280f51a0 0x7f2d281064e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:45.205 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:45.203+0000 7f2d2eee2700 1 -- 172.21.15.174:0/3383272583 >> 172.21.15.174:0/3383272583 conn(0x7f2d280f09a0 msgr2=0x7f2d2805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:45.205 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:45.203+0000 7f2d2eee2700 1 -- 172.21.15.174:0/3383272583 shutdown_connections 2023-12-25T21:11:45.205 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:45.203+0000 7f2d2eee2700 1 -- 172.21.15.174:0/3383272583 wait complete. 2023-12-25T21:11:45.205 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:45.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:45 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:45.202995+0000 mon.smithi061 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.174:0/3383272583' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:46.593 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:46.594 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:46.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:46 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:44.612943+0000 mgr.smithi061.jxxeke (mgr.14164) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:47.683 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.680+0000 7f61e4f35700 1 -- 172.21.15.174:0/282743233 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e00f62a0 msgr2=0x7f61e00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:47.683 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.680+0000 7f61e4f35700 1 --2- 172.21.15.174:0/282743233 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e00f62a0 0x7f61e00f66c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f61d0009fb0 tx=0x7f61d0009b30).stop 2023-12-25T21:11:47.683 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.680+0000 7f61e4f35700 1 -- 172.21.15.174:0/282743233 shutdown_connections 2023-12-25T21:11:47.683 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.680+0000 7f61e4f35700 1 --2- 172.21.15.174:0/282743233 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e00f62a0 0x7f61e00f66c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:47.683 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.680+0000 7f61e4f35700 1 -- 172.21.15.174:0/282743233 >> 172.21.15.174:0/282743233 conn(0x7f61e00f1aa0 msgr2=0x7f61e00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:47.683 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.681+0000 7f61e4f35700 1 -- 172.21.15.174:0/282743233 shutdown_connections 2023-12-25T21:11:47.683 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.681+0000 7f61e4f35700 1 -- 172.21.15.174:0/282743233 wait complete. 2023-12-25T21:11:47.683 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.681+0000 7f61e4f35700 1 Processor -- start 2023-12-25T21:11:47.684 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.681+0000 7f61e4f35700 1 -- start start 2023-12-25T21:11:47.684 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.681+0000 7f61e4f35700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e00f62a0 0x7f61e006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:47.684 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.681+0000 7f61e4f35700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e0069e20 con 0x7f61e00f62a0 2023-12-25T21:11:47.684 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.682+0000 7f61df7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e00f62a0 0x7f61e006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:47.684 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.682+0000 7f61df7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e00f62a0 0x7f61e006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:40938/0 (socket says 172.21.15.174:40938) 2023-12-25T21:11:47.684 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.682+0000 7f61df7fe700 1 -- 172.21.15.174:0/195326181 learned_addr learned my addr 172.21.15.174:0/195326181 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:47.684 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.682+0000 7f61df7fe700 1 -- 172.21.15.174:0/195326181 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d0005040 con 0x7f61e00f62a0 2023-12-25T21:11:47.684 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.682+0000 7f61df7fe700 1 --2- 172.21.15.174:0/195326181 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e00f62a0 0x7f61e006b9a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f61d0013510 tx=0x7f61d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:47.684 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.682+0000 7f61c7fff700 1 -- 172.21.15.174:0/195326181 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61d0009320 con 0x7f61e00f62a0 2023-12-25T21:11:47.684 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.682+0000 7f61e4f35700 1 -- 172.21.15.174:0/195326181 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e0069fc0 con 0x7f61e00f62a0 2023-12-25T21:11:47.684 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.682+0000 7f61c7fff700 1 -- 172.21.15.174:0/195326181 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61d0013f00 con 0x7f61e00f62a0 2023-12-25T21:11:47.684 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.682+0000 7f61c7fff700 1 -- 172.21.15.174:0/195326181 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61d0014dc0 con 0x7f61e00f62a0 2023-12-25T21:11:47.684 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.682+0000 7f61e4f35700 1 -- 172.21.15.174:0/195326181 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e006a450 con 0x7f61e00f62a0 2023-12-25T21:11:47.685 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.683+0000 7f61c7fff700 1 -- 172.21.15.174:0/195326181 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61d0014420 con 0x7f61e00f62a0 2023-12-25T21:11:47.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.683+0000 7f61e4f35700 1 -- 172.21.15.174:0/195326181 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e0059100 con 0x7f61e00f62a0 2023-12-25T21:11:47.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.683+0000 7f61c7fff700 1 --2- 172.21.15.174:0/195326181 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f61c802ff60 0x7f61c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:47.686 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.683+0000 7f61c7fff700 1 -- 172.21.15.174:0/195326181 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61d0041050 con 0x7f61e00f62a0 2023-12-25T21:11:47.687 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.683+0000 7f61deffd700 1 --2- 172.21.15.174:0/195326181 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f61c802ff60 0x7f61c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:47.687 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.684+0000 7f61deffd700 1 --2- 172.21.15.174:0/195326181 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f61c802ff60 0x7f61c8032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f61d40096a0 tx=0x7f61d4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:47.689 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.688+0000 7f61c7fff700 1 -- 172.21.15.174:0/195326181 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61d00095f0 con 0x7f61e00f62a0 2023-12-25T21:11:47.894 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.893+0000 7f61e4f35700 1 -- 172.21.15.174:0/195326181 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61e00fbce0 con 0x7f61e00f62a0 2023-12-25T21:11:47.894 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.894+0000 7f61c7fff700 1 -- 172.21.15.174:0/195326181 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f61d001cb10 con 0x7f61e00f62a0 2023-12-25T21:11:47.895 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:47.895 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:47.896 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.896+0000 7f61e4f35700 1 -- 172.21.15.174:0/195326181 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f61c802ff60 msgr2=0x7f61c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:47.896 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.896+0000 7f61e4f35700 1 --2- 172.21.15.174:0/195326181 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f61c802ff60 0x7f61c8032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f61d40096a0 tx=0x7f61d4006b40).stop 2023-12-25T21:11:47.897 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.896+0000 7f61e4f35700 1 -- 172.21.15.174:0/195326181 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e00f62a0 msgr2=0x7f61e006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:47.897 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.896+0000 7f61e4f35700 1 --2- 172.21.15.174:0/195326181 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e00f62a0 0x7f61e006b9a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f61d0013510 tx=0x7f61d000b040).stop 2023-12-25T21:11:47.897 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.896+0000 7f61e4f35700 1 -- 172.21.15.174:0/195326181 shutdown_connections 2023-12-25T21:11:47.897 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.896+0000 7f61e4f35700 1 --2- 172.21.15.174:0/195326181 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f61c802ff60 0x7f61c8032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:47.897 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.896+0000 7f61e4f35700 1 --2- 172.21.15.174:0/195326181 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e00f62a0 0x7f61e006b9a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:47.897 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.896+0000 7f61e4f35700 1 -- 172.21.15.174:0/195326181 >> 172.21.15.174:0/195326181 conn(0x7f61e00f1aa0 msgr2=0x7f61e00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:47.897 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.896+0000 7f61e4f35700 1 -- 172.21.15.174:0/195326181 shutdown_connections 2023-12-25T21:11:47.897 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:47.896+0000 7f61e4f35700 1 -- 172.21.15.174:0/195326181 wait complete. 2023-12-25T21:11:47.897 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:48.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:48 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:46.613178+0000 mgr.smithi061.jxxeke (mgr.14164) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:48.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:48 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:47.895281+0000 mon.smithi061 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.174:0/195326181' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:49.295 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:49.296 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:50.495 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.493+0000 7f58f8aed700 1 -- 172.21.15.174:0/1804713389 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58f40f51a0 msgr2=0x7f58f40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:50.495 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.493+0000 7f58f8aed700 1 --2- 172.21.15.174:0/1804713389 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58f40f51a0 0x7f58f40f55c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f58e4009fb0 tx=0x7f58e4009b30).stop 2023-12-25T21:11:50.495 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.493+0000 7f58f8aed700 1 -- 172.21.15.174:0/1804713389 shutdown_connections 2023-12-25T21:11:50.495 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.493+0000 7f58f8aed700 1 --2- 172.21.15.174:0/1804713389 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58f40f51a0 0x7f58f40f55c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:50.495 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.493+0000 7f58f8aed700 1 -- 172.21.15.174:0/1804713389 >> 172.21.15.174:0/1804713389 conn(0x7f58f40f09a0 msgr2=0x7f58f40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:50.495 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.493+0000 7f58f8aed700 1 -- 172.21.15.174:0/1804713389 shutdown_connections 2023-12-25T21:11:50.495 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.493+0000 7f58f8aed700 1 -- 172.21.15.174:0/1804713389 wait complete. 2023-12-25T21:11:50.496 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.493+0000 7f58f8aed700 1 Processor -- start 2023-12-25T21:11:50.496 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.494+0000 7f58f8aed700 1 -- start start 2023-12-25T21:11:50.496 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.494+0000 7f58f8aed700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58f40f51a0 0x7f58f41091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:50.496 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.494+0000 7f58f8aed700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f4060240 con 0x7f58f40f51a0 2023-12-25T21:11:50.496 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.494+0000 7f58f37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58f40f51a0 0x7f58f41091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:50.496 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.494+0000 7f58f37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58f40f51a0 0x7f58f41091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:40954/0 (socket says 172.21.15.174:40954) 2023-12-25T21:11:50.496 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.494+0000 7f58f37fe700 1 -- 172.21.15.174:0/545084373 learned_addr learned my addr 172.21.15.174:0/545084373 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:50.496 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.494+0000 7f58f37fe700 1 -- 172.21.15.174:0/545084373 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e4005040 con 0x7f58f40f51a0 2023-12-25T21:11:50.496 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.494+0000 7f58f37fe700 1 --2- 172.21.15.174:0/545084373 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58f40f51a0 0x7f58f41091b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f58e4013510 tx=0x7f58e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:50.497 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.495+0000 7f58dbfff700 1 -- 172.21.15.174:0/545084373 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58e4009320 con 0x7f58f40f51a0 2023-12-25T21:11:50.498 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.495+0000 7f58f8aed700 1 -- 172.21.15.174:0/545084373 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f4060440 con 0x7f58f40f51a0 2023-12-25T21:11:50.498 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.495+0000 7f58dbfff700 1 -- 172.21.15.174:0/545084373 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f58e4013f00 con 0x7f58f40f51a0 2023-12-25T21:11:50.498 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.495+0000 7f58dbfff700 1 -- 172.21.15.174:0/545084373 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58e4014dc0 con 0x7f58f40f51a0 2023-12-25T21:11:50.498 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.495+0000 7f58f8aed700 1 -- 172.21.15.174:0/545084373 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f4109860 con 0x7f58f40f51a0 2023-12-25T21:11:50.498 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.495+0000 7f58dbfff700 1 -- 172.21.15.174:0/545084373 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f58e401d3d0 con 0x7f58f40f51a0 2023-12-25T21:11:50.498 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.496+0000 7f58dbfff700 1 --2- 172.21.15.174:0/545084373 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f58dc02ff60 0x7f58dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:50.498 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.496+0000 7f58dbfff700 1 -- 172.21.15.174:0/545084373 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f58e40410e0 con 0x7f58f40f51a0 2023-12-25T21:11:50.498 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.496+0000 7f58f8aed700 1 -- 172.21.15.174:0/545084373 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f4045ad0 con 0x7f58f40f51a0 2023-12-25T21:11:50.498 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.496+0000 7f58f2ffd700 1 --2- 172.21.15.174:0/545084373 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f58dc02ff60 0x7f58dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:50.498 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.496+0000 7f58f2ffd700 1 --2- 172.21.15.174:0/545084373 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f58dc02ff60 0x7f58dc032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f58e80096a0 tx=0x7f58e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:50.501 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.501+0000 7f58dbfff700 1 -- 172.21.15.174:0/545084373 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f58e4012920 con 0x7f58f40f51a0 2023-12-25T21:11:50.703 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.702+0000 7f58f8aed700 1 -- 172.21.15.174:0/545084373 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58f4109f40 con 0x7f58f40f51a0 2023-12-25T21:11:50.704 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.703+0000 7f58dbfff700 1 -- 172.21.15.174:0/545084373 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f58e402b020 con 0x7f58f40f51a0 2023-12-25T21:11:50.704 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:50.704 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:50.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:50 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:48.613524+0000 mgr.smithi061.jxxeke (mgr.14164) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:50.706 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.705+0000 7f58f8aed700 1 -- 172.21.15.174:0/545084373 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f58dc02ff60 msgr2=0x7f58dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:50.706 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.705+0000 7f58f8aed700 1 --2- 172.21.15.174:0/545084373 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f58dc02ff60 0x7f58dc032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f58e80096a0 tx=0x7f58e8006b40).stop 2023-12-25T21:11:50.706 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.705+0000 7f58f8aed700 1 -- 172.21.15.174:0/545084373 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58f40f51a0 msgr2=0x7f58f41091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:50.706 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.705+0000 7f58f8aed700 1 --2- 172.21.15.174:0/545084373 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58f40f51a0 0x7f58f41091b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f58e4013510 tx=0x7f58e400b040).stop 2023-12-25T21:11:50.706 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.705+0000 7f58f8aed700 1 -- 172.21.15.174:0/545084373 shutdown_connections 2023-12-25T21:11:50.706 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.705+0000 7f58f8aed700 1 --2- 172.21.15.174:0/545084373 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f58dc02ff60 0x7f58dc032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:50.706 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.705+0000 7f58f8aed700 1 --2- 172.21.15.174:0/545084373 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58f40f51a0 0x7f58f41091b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:50.706 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.705+0000 7f58f8aed700 1 -- 172.21.15.174:0/545084373 >> 172.21.15.174:0/545084373 conn(0x7f58f40f09a0 msgr2=0x7f58f40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:50.707 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.705+0000 7f58f8aed700 1 -- 172.21.15.174:0/545084373 shutdown_connections 2023-12-25T21:11:50.707 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:50.705+0000 7f58f8aed700 1 -- 172.21.15.174:0/545084373 wait complete. 2023-12-25T21:11:50.707 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:51.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:51 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:50.704723+0000 mon.smithi061 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.174:0/545084373' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:52.092 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:52.093 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:52.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:52 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:50.613762+0000 mgr.smithi061.jxxeke (mgr.14164) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:53.209 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.208+0000 7f520d635700 1 -- 172.21.15.174:0/1444583838 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52080f5410 msgr2=0x7f52080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:53.209 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.208+0000 7f520d635700 1 --2- 172.21.15.174:0/1444583838 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52080f5410 0x7f52080f5830 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f51fc009fb0 tx=0x7f51fc009b30).stop 2023-12-25T21:11:53.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.208+0000 7f520d635700 1 -- 172.21.15.174:0/1444583838 shutdown_connections 2023-12-25T21:11:53.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.208+0000 7f520d635700 1 --2- 172.21.15.174:0/1444583838 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52080f5410 0x7f52080f5830 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:53.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.208+0000 7f520d635700 1 -- 172.21.15.174:0/1444583838 >> 172.21.15.174:0/1444583838 conn(0x7f52080f0b70 msgr2=0x7f52080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:53.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.208+0000 7f520d635700 1 -- 172.21.15.174:0/1444583838 shutdown_connections 2023-12-25T21:11:53.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.208+0000 7f520d635700 1 -- 172.21.15.174:0/1444583838 wait complete. 2023-12-25T21:11:53.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.209+0000 7f520d635700 1 Processor -- start 2023-12-25T21:11:53.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.209+0000 7f520d635700 1 -- start start 2023-12-25T21:11:53.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.209+0000 7f520d635700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52080f5410 0x7f5208109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:53.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.209+0000 7f520d635700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5208060220 con 0x7f52080f5410 2023-12-25T21:11:53.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.209+0000 7f5206ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52080f5410 0x7f5208109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:53.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.209+0000 7f5206ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52080f5410 0x7f5208109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:40966/0 (socket says 172.21.15.174:40966) 2023-12-25T21:11:53.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.209+0000 7f5206ffd700 1 -- 172.21.15.174:0/3803737231 learned_addr learned my addr 172.21.15.174:0/3803737231 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:53.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.210+0000 7f5206ffd700 1 -- 172.21.15.174:0/3803737231 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51fc005040 con 0x7f52080f5410 2023-12-25T21:11:53.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.210+0000 7f5206ffd700 1 --2- 172.21.15.174:0/3803737231 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52080f5410 0x7f5208109450 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f51fc013510 tx=0x7f51fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:53.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.210+0000 7f51ef7fe700 1 -- 172.21.15.174:0/3803737231 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51fc009320 con 0x7f52080f5410 2023-12-25T21:11:53.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.210+0000 7f520d635700 1 -- 172.21.15.174:0/3803737231 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5208060420 con 0x7f52080f5410 2023-12-25T21:11:53.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.210+0000 7f51ef7fe700 1 -- 172.21.15.174:0/3803737231 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f51fc013f00 con 0x7f52080f5410 2023-12-25T21:11:53.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.210+0000 7f51ef7fe700 1 -- 172.21.15.174:0/3803737231 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51fc014d90 con 0x7f52080f5410 2023-12-25T21:11:53.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.210+0000 7f520d635700 1 -- 172.21.15.174:0/3803737231 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5208109b00 con 0x7f52080f5410 2023-12-25T21:11:53.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.211+0000 7f51ef7fe700 1 -- 172.21.15.174:0/3803737231 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51fc0263a0 con 0x7f52080f5410 2023-12-25T21:11:53.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.211+0000 7f51ef7fe700 1 --2- 172.21.15.174:0/3803737231 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f51f002ff60 0x7f51f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:53.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.211+0000 7f51ef7fe700 1 -- 172.21.15.174:0/3803737231 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f51fc0410c0 con 0x7f52080f5410 2023-12-25T21:11:53.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.211+0000 7f520d635700 1 -- 172.21.15.174:0/3803737231 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5208059100 con 0x7f52080f5410 2023-12-25T21:11:53.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.211+0000 7f52067fc700 1 --2- 172.21.15.174:0/3803737231 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f51f002ff60 0x7f51f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:53.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.211+0000 7f52067fc700 1 --2- 172.21.15.174:0/3803737231 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f51f002ff60 0x7f51f0032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f51f80096a0 tx=0x7f51f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:53.216 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.216+0000 7f51ef7fe700 1 -- 172.21.15.174:0/3803737231 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51fc012920 con 0x7f52080f5410 2023-12-25T21:11:53.423 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.422+0000 7f520d635700 1 -- 172.21.15.174:0/3803737231 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f520810a250 con 0x7f52080f5410 2023-12-25T21:11:53.423 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.423+0000 7f51ef7fe700 1 -- 172.21.15.174:0/3803737231 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f51fc02b020 con 0x7f52080f5410 2023-12-25T21:11:53.424 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:53.424 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:53.425 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.425+0000 7f520d635700 1 -- 172.21.15.174:0/3803737231 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f51f002ff60 msgr2=0x7f51f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:53.425 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.425+0000 7f520d635700 1 --2- 172.21.15.174:0/3803737231 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f51f002ff60 0x7f51f0032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f51f80096a0 tx=0x7f51f8006b40).stop 2023-12-25T21:11:53.425 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.425+0000 7f520d635700 1 -- 172.21.15.174:0/3803737231 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52080f5410 msgr2=0x7f5208109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:53.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.425+0000 7f520d635700 1 --2- 172.21.15.174:0/3803737231 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52080f5410 0x7f5208109450 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f51fc013510 tx=0x7f51fc00b040).stop 2023-12-25T21:11:53.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.425+0000 7f520d635700 1 -- 172.21.15.174:0/3803737231 shutdown_connections 2023-12-25T21:11:53.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.425+0000 7f520d635700 1 --2- 172.21.15.174:0/3803737231 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f51f002ff60 0x7f51f0032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:53.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.425+0000 7f520d635700 1 --2- 172.21.15.174:0/3803737231 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52080f5410 0x7f5208109450 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:53.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.425+0000 7f520d635700 1 -- 172.21.15.174:0/3803737231 >> 172.21.15.174:0/3803737231 conn(0x7f52080f0b70 msgr2=0x7f52080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:53.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.425+0000 7f520d635700 1 -- 172.21.15.174:0/3803737231 shutdown_connections 2023-12-25T21:11:53.426 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:53.425+0000 7f520d635700 1 -- 172.21.15.174:0/3803737231 wait complete. 2023-12-25T21:11:53.426 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:53.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:52.614092+0000 mgr.smithi061.jxxeke (mgr.14164) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:53.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:52.706206+0000 mon.smithi061 (mon.0) 223 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/mirror_snapshot_schedule"}]: dispatch 2023-12-25T21:11:53.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:52.707171+0000 mon.smithi061 (mon.0) 224 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/trash_purge_schedule"}]: dispatch 2023-12-25T21:11:54.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:54 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:53.424455+0000 mon.smithi061 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.174:0/3803737231' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:54.822 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:54.824 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:55.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:55 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:54.614431+0000 mgr.smithi061.jxxeke (mgr.14164) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:56.038 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.034+0000 7fd605d2e700 1 -- 172.21.15.174:0/3399237855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6000f51a0 msgr2=0x7fd6000f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:56.039 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.034+0000 7fd605d2e700 1 --2- 172.21.15.174:0/3399237855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6000f51a0 0x7fd6000f55c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fd5f4009fb0 tx=0x7fd5f4009b30).stop 2023-12-25T21:11:56.039 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.034+0000 7fd605d2e700 1 -- 172.21.15.174:0/3399237855 shutdown_connections 2023-12-25T21:11:56.039 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.034+0000 7fd605d2e700 1 --2- 172.21.15.174:0/3399237855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6000f51a0 0x7fd6000f55c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:56.039 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.034+0000 7fd605d2e700 1 -- 172.21.15.174:0/3399237855 >> 172.21.15.174:0/3399237855 conn(0x7fd6000f09a0 msgr2=0x7fd6000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:56.039 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.034+0000 7fd605d2e700 1 -- 172.21.15.174:0/3399237855 shutdown_connections 2023-12-25T21:11:56.039 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.034+0000 7fd605d2e700 1 -- 172.21.15.174:0/3399237855 wait complete. 2023-12-25T21:11:56.040 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.034+0000 7fd605d2e700 1 Processor -- start 2023-12-25T21:11:56.040 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.034+0000 7fd605d2e700 1 -- start start 2023-12-25T21:11:56.040 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.035+0000 7fd605d2e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6000f51a0 0x7fd6001064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:56.040 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.035+0000 7fd605d2e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd600106990 con 0x7fd6000f51a0 2023-12-25T21:11:56.040 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.035+0000 7fd604d2c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6000f51a0 0x7fd6001064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:56.041 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.035+0000 7fd604d2c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6000f51a0 0x7fd6001064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:40990/0 (socket says 172.21.15.174:40990) 2023-12-25T21:11:56.041 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.035+0000 7fd604d2c700 1 -- 172.21.15.174:0/142494418 learned_addr learned my addr 172.21.15.174:0/142494418 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:56.041 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.035+0000 7fd604d2c700 1 -- 172.21.15.174:0/142494418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f4005040 con 0x7fd6000f51a0 2023-12-25T21:11:56.041 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.035+0000 7fd604d2c700 1 --2- 172.21.15.174:0/142494418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6000f51a0 0x7fd6001064a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fd5f4013510 tx=0x7fd5f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:56.041 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.036+0000 7fd5fd7fa700 1 -- 172.21.15.174:0/142494418 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5f4009320 con 0x7fd6000f51a0 2023-12-25T21:11:56.041 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.036+0000 7fd605d2e700 1 -- 172.21.15.174:0/142494418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd600106b90 con 0x7fd6000f51a0 2023-12-25T21:11:56.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.036+0000 7fd5fd7fa700 1 -- 172.21.15.174:0/142494418 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd5f4013f00 con 0x7fd6000f51a0 2023-12-25T21:11:56.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.036+0000 7fd5fd7fa700 1 -- 172.21.15.174:0/142494418 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5f4014dc0 con 0x7fd6000f51a0 2023-12-25T21:11:56.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.036+0000 7fd605d2e700 1 -- 172.21.15.174:0/142494418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd600107020 con 0x7fd6000f51a0 2023-12-25T21:11:56.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.037+0000 7fd5fd7fa700 1 -- 172.21.15.174:0/142494418 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5f4014420 con 0x7fd6000f51a0 2023-12-25T21:11:56.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.037+0000 7fd605d2e700 1 -- 172.21.15.174:0/142494418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd600045ad0 con 0x7fd6000f51a0 2023-12-25T21:11:56.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.037+0000 7fd5fd7fa700 1 --2- 172.21.15.174:0/142494418 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd5f002ff60 0x7fd5f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:56.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.037+0000 7fd5fd7fa700 1 -- 172.21.15.174:0/142494418 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd5f4041050 con 0x7fd6000f51a0 2023-12-25T21:11:56.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.037+0000 7fd5fffff700 1 --2- 172.21.15.174:0/142494418 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd5f002ff60 0x7fd5f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:56.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.037+0000 7fd5fffff700 1 --2- 172.21.15.174:0/142494418 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd5f002ff60 0x7fd5f0032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fd5ec0096a0 tx=0x7fd5ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:56.044 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.042+0000 7fd5fd7fa700 1 -- 172.21.15.174:0/142494418 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd5f40095f0 con 0x7fd6000f51a0 2023-12-25T21:11:56.248 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.247+0000 7fd605d2e700 1 -- 172.21.15.174:0/142494418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd600107ed0 con 0x7fd6000f51a0 2023-12-25T21:11:56.248 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.247+0000 7fd5fd7fa700 1 -- 172.21.15.174:0/142494418 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd5f401cb10 con 0x7fd6000f51a0 2023-12-25T21:11:56.249 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:56.249 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:56.250 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.249+0000 7fd605d2e700 1 -- 172.21.15.174:0/142494418 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd5f002ff60 msgr2=0x7fd5f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:56.250 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.249+0000 7fd605d2e700 1 --2- 172.21.15.174:0/142494418 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd5f002ff60 0x7fd5f0032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fd5ec0096a0 tx=0x7fd5ec006b40).stop 2023-12-25T21:11:56.250 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.249+0000 7fd605d2e700 1 -- 172.21.15.174:0/142494418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6000f51a0 msgr2=0x7fd6001064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:56.250 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.249+0000 7fd605d2e700 1 --2- 172.21.15.174:0/142494418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6000f51a0 0x7fd6001064a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fd5f4013510 tx=0x7fd5f400b040).stop 2023-12-25T21:11:56.250 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.249+0000 7fd605d2e700 1 -- 172.21.15.174:0/142494418 shutdown_connections 2023-12-25T21:11:56.250 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.249+0000 7fd605d2e700 1 --2- 172.21.15.174:0/142494418 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd5f002ff60 0x7fd5f0032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:56.250 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.249+0000 7fd605d2e700 1 --2- 172.21.15.174:0/142494418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6000f51a0 0x7fd6001064a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:56.250 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.249+0000 7fd605d2e700 1 -- 172.21.15.174:0/142494418 >> 172.21.15.174:0/142494418 conn(0x7fd6000f09a0 msgr2=0x7fd60005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:56.251 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.249+0000 7fd605d2e700 1 -- 172.21.15.174:0/142494418 shutdown_connections 2023-12-25T21:11:56.251 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:56.249+0000 7fd605d2e700 1 -- 172.21.15.174:0/142494418 wait complete. 2023-12-25T21:11:56.251 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:56.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:56 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:56.248908+0000 mon.smithi061 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.174:0/142494418' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:11:57.666 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:11:57.666 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:11:57.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:57 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:56.614767+0000 mgr.smithi061.jxxeke (mgr.14164) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:58.836 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.835+0000 7f626af8b700 1 -- 172.21.15.174:0/2001943829 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6264060280 msgr2=0x7f62640f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:58.837 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.835+0000 7f626af8b700 1 --2- 172.21.15.174:0/2001943829 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6264060280 0x7f62640f90f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f6258009fb0 tx=0x7f6258009b30).stop 2023-12-25T21:11:58.837 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.835+0000 7f626af8b700 1 -- 172.21.15.174:0/2001943829 shutdown_connections 2023-12-25T21:11:58.837 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.835+0000 7f626af8b700 1 --2- 172.21.15.174:0/2001943829 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6264060280 0x7f62640f90f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:58.837 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.835+0000 7f626af8b700 1 -- 172.21.15.174:0/2001943829 >> 172.21.15.174:0/2001943829 conn(0x7f62640f0b70 msgr2=0x7f62640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:58.837 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.835+0000 7f626af8b700 1 -- 172.21.15.174:0/2001943829 shutdown_connections 2023-12-25T21:11:58.837 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.835+0000 7f626af8b700 1 -- 172.21.15.174:0/2001943829 wait complete. 2023-12-25T21:11:58.837 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.836+0000 7f626af8b700 1 Processor -- start 2023-12-25T21:11:58.837 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.836+0000 7f626af8b700 1 -- start start 2023-12-25T21:11:58.837 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.836+0000 7f626af8b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6264060280 0x7f62641066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:58.837 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.836+0000 7f626af8b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6264106bd0 con 0x7f6264060280 2023-12-25T21:11:58.839 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.836+0000 7f6268d27700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6264060280 0x7f62641066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:58.839 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.836+0000 7f6268d27700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6264060280 0x7f62641066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:53162/0 (socket says 172.21.15.174:53162) 2023-12-25T21:11:58.839 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.836+0000 7f6268d27700 1 -- 172.21.15.174:0/3224015996 learned_addr learned my addr 172.21.15.174:0/3224015996 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:11:58.839 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.837+0000 7f6268d27700 1 -- 172.21.15.174:0/3224015996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6258005040 con 0x7f6264060280 2023-12-25T21:11:58.839 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.837+0000 7f6268d27700 1 --2- 172.21.15.174:0/3224015996 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6264060280 0x7f62641066e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f6258013510 tx=0x7f625800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:58.839 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.837+0000 7f62617fa700 1 -- 172.21.15.174:0/3224015996 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6258009320 con 0x7f6264060280 2023-12-25T21:11:58.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.837+0000 7f626af8b700 1 -- 172.21.15.174:0/3224015996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6264106dd0 con 0x7f6264060280 2023-12-25T21:11:58.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.837+0000 7f62617fa700 1 -- 172.21.15.174:0/3224015996 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6258013f00 con 0x7f6264060280 2023-12-25T21:11:58.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.837+0000 7f62617fa700 1 -- 172.21.15.174:0/3224015996 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6258014dc0 con 0x7f6264060280 2023-12-25T21:11:58.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.837+0000 7f626af8b700 1 -- 172.21.15.174:0/3224015996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6264107260 con 0x7f6264060280 2023-12-25T21:11:58.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.838+0000 7f62617fa700 1 -- 172.21.15.174:0/3224015996 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f625801d3d0 con 0x7f6264060280 2023-12-25T21:11:58.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.838+0000 7f62617fa700 1 --2- 172.21.15.174:0/3224015996 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f625402ff60 0x7f6254032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:11:58.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.838+0000 7f62617fa700 1 -- 172.21.15.174:0/3224015996 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f62580410c0 con 0x7f6264060280 2023-12-25T21:11:58.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.838+0000 7f626af8b700 1 -- 172.21.15.174:0/3224015996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6264045ad0 con 0x7f6264060280 2023-12-25T21:11:58.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.838+0000 7f6263fff700 1 --2- 172.21.15.174:0/3224015996 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f625402ff60 0x7f6254032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:11:58.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.839+0000 7f6263fff700 1 --2- 172.21.15.174:0/3224015996 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f625402ff60 0x7f6254032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f62500096a0 tx=0x7f6250006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:11:58.844 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:58.843+0000 7f62617fa700 1 -- 172.21.15.174:0/3224015996 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6258026490 con 0x7f6264060280 2023-12-25T21:11:59.051 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:59.050+0000 7f626af8b700 1 -- 172.21.15.174:0/3224015996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6264107fa0 con 0x7f6264060280 2023-12-25T21:11:59.051 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:59.051+0000 7f62617fa700 1 -- 172.21.15.174:0/3224015996 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f62580126f0 con 0x7f6264060280 2023-12-25T21:11:59.052 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:11:59.052 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:11:59.053 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:59.052+0000 7f626af8b700 1 -- 172.21.15.174:0/3224015996 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f625402ff60 msgr2=0x7f6254032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:59.053 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:59.052+0000 7f626af8b700 1 --2- 172.21.15.174:0/3224015996 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f625402ff60 0x7f6254032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f62500096a0 tx=0x7f6250006b40).stop 2023-12-25T21:11:59.053 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:59.052+0000 7f626af8b700 1 -- 172.21.15.174:0/3224015996 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6264060280 msgr2=0x7f62641066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:11:59.053 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:59.052+0000 7f626af8b700 1 --2- 172.21.15.174:0/3224015996 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6264060280 0x7f62641066e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f6258013510 tx=0x7f625800b040).stop 2023-12-25T21:11:59.054 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:59.052+0000 7f626af8b700 1 -- 172.21.15.174:0/3224015996 shutdown_connections 2023-12-25T21:11:59.054 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:59.052+0000 7f626af8b700 1 --2- 172.21.15.174:0/3224015996 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f625402ff60 0x7f6254032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:59.054 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:59.052+0000 7f626af8b700 1 --2- 172.21.15.174:0/3224015996 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6264060280 0x7f62641066e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:11:59.054 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:59.052+0000 7f626af8b700 1 -- 172.21.15.174:0/3224015996 >> 172.21.15.174:0/3224015996 conn(0x7f62640f0b70 msgr2=0x7f62640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:11:59.054 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:59.053+0000 7f626af8b700 1 -- 172.21.15.174:0/3224015996 shutdown_connections 2023-12-25T21:11:59.054 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:11:59.053+0000 7f626af8b700 1 -- 172.21.15.174:0/3224015996 wait complete. 2023-12-25T21:11:59.054 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:11:59.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:59 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:11:58.615119+0000 mgr.smithi061.jxxeke (mgr.14164) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:11:59.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:11:59 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:11:59.052232+0000 mon.smithi061 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.174:0/3224015996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:00.445 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:00.446 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:01.633 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.631+0000 7f5d72863700 1 -- 172.21.15.174:0/2535519104 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d6c0f6510 msgr2=0x7f5d6c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:01.633 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.631+0000 7f5d72863700 1 --2- 172.21.15.174:0/2535519104 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d6c0f6510 0x7f5d6c0f6930 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f5d5c009fb0 tx=0x7f5d5c009b30).stop 2023-12-25T21:12:01.633 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.631+0000 7f5d72863700 1 -- 172.21.15.174:0/2535519104 shutdown_connections 2023-12-25T21:12:01.633 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.631+0000 7f5d72863700 1 --2- 172.21.15.174:0/2535519104 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d6c0f6510 0x7f5d6c0f6930 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:01.633 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.631+0000 7f5d72863700 1 -- 172.21.15.174:0/2535519104 >> 172.21.15.174:0/2535519104 conn(0x7f5d6c0f1c70 msgr2=0x7f5d6c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:01.633 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.631+0000 7f5d72863700 1 -- 172.21.15.174:0/2535519104 shutdown_connections 2023-12-25T21:12:01.633 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.631+0000 7f5d72863700 1 -- 172.21.15.174:0/2535519104 wait complete. 2023-12-25T21:12:01.633 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.631+0000 7f5d72863700 1 Processor -- start 2023-12-25T21:12:01.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.631+0000 7f5d72863700 1 -- start start 2023-12-25T21:12:01.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.632+0000 7f5d72863700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d6c0f6510 0x7f5d6c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:01.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.632+0000 7f5d72863700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c106c20 con 0x7f5d6c0f6510 2023-12-25T21:12:01.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.632+0000 7f5d6bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d6c0f6510 0x7f5d6c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:01.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.632+0000 7f5d6bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d6c0f6510 0x7f5d6c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:53190/0 (socket says 172.21.15.174:53190) 2023-12-25T21:12:01.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.632+0000 7f5d6bfff700 1 -- 172.21.15.174:0/4243320381 learned_addr learned my addr 172.21.15.174:0/4243320381 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:01.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.632+0000 7f5d6bfff700 1 -- 172.21.15.174:0/4243320381 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5c005040 con 0x7f5d6c0f6510 2023-12-25T21:12:01.635 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.633+0000 7f5d6bfff700 1 --2- 172.21.15.174:0/4243320381 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d6c0f6510 0x7f5d6c106730 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f5d5c013510 tx=0x7f5d5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:01.635 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.633+0000 7f5d68ff9700 1 -- 172.21.15.174:0/4243320381 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d5c009320 con 0x7f5d6c0f6510 2023-12-25T21:12:01.635 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.633+0000 7f5d72863700 1 -- 172.21.15.174:0/4243320381 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d6c106e20 con 0x7f5d6c0f6510 2023-12-25T21:12:01.635 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.633+0000 7f5d68ff9700 1 -- 172.21.15.174:0/4243320381 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d5c013f00 con 0x7f5d6c0f6510 2023-12-25T21:12:01.635 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.633+0000 7f5d68ff9700 1 -- 172.21.15.174:0/4243320381 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d5c014d90 con 0x7f5d6c0f6510 2023-12-25T21:12:01.636 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.633+0000 7f5d72863700 1 -- 172.21.15.174:0/4243320381 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d6c1072b0 con 0x7f5d6c0f6510 2023-12-25T21:12:01.636 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.634+0000 7f5d68ff9700 1 -- 172.21.15.174:0/4243320381 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d5c014420 con 0x7f5d6c0f6510 2023-12-25T21:12:01.636 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.634+0000 7f5d72863700 1 -- 172.21.15.174:0/4243320381 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d6c045ad0 con 0x7f5d6c0f6510 2023-12-25T21:12:01.636 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.634+0000 7f5d68ff9700 1 --2- 172.21.15.174:0/4243320381 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5d5402ff60 0x7f5d54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:01.636 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.634+0000 7f5d68ff9700 1 -- 172.21.15.174:0/4243320381 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d5c0410c0 con 0x7f5d6c0f6510 2023-12-25T21:12:01.636 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.634+0000 7f5d6b7fe700 1 --2- 172.21.15.174:0/4243320381 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5d5402ff60 0x7f5d54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:01.636 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.634+0000 7f5d6b7fe700 1 --2- 172.21.15.174:0/4243320381 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5d5402ff60 0x7f5d54032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f5d600096a0 tx=0x7f5d60006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:01.639 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.639+0000 7f5d68ff9700 1 -- 172.21.15.174:0/4243320381 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d5c0095f0 con 0x7f5d6c0f6510 2023-12-25T21:12:01.845 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.844+0000 7f5d72863700 1 -- 172.21.15.174:0/4243320381 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d6c0fbf60 con 0x7f5d6c0f6510 2023-12-25T21:12:01.846 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.845+0000 7f5d68ff9700 1 -- 172.21.15.174:0/4243320381 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5d5c01db70 con 0x7f5d6c0f6510 2023-12-25T21:12:01.846 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:01.846 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:01.847 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.847+0000 7f5d72863700 1 -- 172.21.15.174:0/4243320381 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5d5402ff60 msgr2=0x7f5d54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:01.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.847+0000 7f5d72863700 1 --2- 172.21.15.174:0/4243320381 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5d5402ff60 0x7f5d54032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f5d600096a0 tx=0x7f5d60006b40).stop 2023-12-25T21:12:01.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.847+0000 7f5d72863700 1 -- 172.21.15.174:0/4243320381 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d6c0f6510 msgr2=0x7f5d6c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:01.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.847+0000 7f5d72863700 1 --2- 172.21.15.174:0/4243320381 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d6c0f6510 0x7f5d6c106730 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f5d5c013510 tx=0x7f5d5c00b040).stop 2023-12-25T21:12:01.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.847+0000 7f5d72863700 1 -- 172.21.15.174:0/4243320381 shutdown_connections 2023-12-25T21:12:01.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.847+0000 7f5d72863700 1 --2- 172.21.15.174:0/4243320381 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5d5402ff60 0x7f5d54032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:01.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.847+0000 7f5d72863700 1 --2- 172.21.15.174:0/4243320381 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d6c0f6510 0x7f5d6c106730 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:01.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.847+0000 7f5d72863700 1 -- 172.21.15.174:0/4243320381 >> 172.21.15.174:0/4243320381 conn(0x7f5d6c0f1c70 msgr2=0x7f5d6c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:01.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.847+0000 7f5d72863700 1 -- 172.21.15.174:0/4243320381 shutdown_connections 2023-12-25T21:12:01.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:01.847+0000 7f5d72863700 1 -- 172.21.15.174:0/4243320381 wait complete. 2023-12-25T21:12:01.849 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:02.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:02 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:00.615411+0000 mgr.smithi061.jxxeke (mgr.14164) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:02.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:02 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:01.846614+0000 mon.smithi061 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.174:0/4243320381' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:03.263 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:03.263 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:03.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cephadm 2023-12-25T21:12:02.364076+0000 mgr.smithi061.jxxeke (mgr.14164) 156 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi061 ... 2023-12-25T21:12:03.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-12-25T21:12:03.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-25T21:12:03.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr time="2023-12-25T21:07:41Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T21:12:03.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr time="2023-12-25T21:08:18Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T21:12:03.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr time="2023-12-25T21:08:55Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T21:12:03.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-25T21:12:03.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-12-25T21:12:03.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-25T21:12:03.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr time="2023-12-25T21:10:09Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T21:12:03.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr time="2023-12-25T21:10:47Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T21:12:03.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr time="2023-12-25T21:11:24Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T21:12:03.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-25T21:12:03.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Traceback (most recent call last): 2023-12-25T21:12:03.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-25T21:12:03.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: main() 2023-12-25T21:12:03.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-25T21:12:03.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: r = ctx.func(ctx) 2023-12-25T21:12:03.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-25T21:12:03.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: return func(ctx) 2023-12-25T21:12:03.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-25T21:12:03.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-25T21:12:03.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-12-25T21:12:03.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-12-25T21:12:03.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-25T21:12:03.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: raise RuntimeError('uid/gid not found') 2023-12-25T21:12:03.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: RuntimeError: uid/gid not found 2023-12-25T21:12:03.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Traceback (most recent call last): 2023-12-25T21:12:03.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-25T21:12:03.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: yield (conn, connr) 2023-12-25T21:12:03.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-25T21:12:03.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: code, '\n'.join(err))) 2023-12-25T21:12:03.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi061 ... 2023-12-25T21:12:03.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-12-25T21:12:03.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-25T21:12:03.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr time="2023-12-25T21:07:41Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T21:12:03.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr time="2023-12-25T21:08:18Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T21:12:03.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr time="2023-12-25T21:08:55Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T21:12:03.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-25T21:12:03.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi061 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-12-25T21:12:03.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-25T21:12:03.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr time="2023-12-25T21:10:09Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T21:12:03.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr time="2023-12-25T21:10:47Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T21:12:03.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr time="2023-12-25T21:11:24Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-25T21:12:03.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-25T21:12:03.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Traceback (most recent call last): 2023-12-25T21:12:03.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-25T21:12:03.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: main() 2023-12-25T21:12:03.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-25T21:12:03.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: r = ctx.func(ctx) 2023-12-25T21:12:03.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-25T21:12:03.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: return func(ctx) 2023-12-25T21:12:03.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-25T21:12:03.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-25T21:12:03.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-12-25T21:12:03.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-12-25T21:12:03.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-25T21:12:03.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: raise RuntimeError('uid/gid not found') 2023-12-25T21:12:03.711 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: RuntimeError: uid/gid not found 2023-12-25T21:12:03.711 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:02.365367+0000 mon.smithi061 (mon.0) 229 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi061", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-25T21:12:03.711 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:02.368450+0000 mon.smithi061 (mon.0) 230 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi061", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-25T21:12:03.711 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:02.369359+0000 mon.smithi061 (mon.0) 231 : audit [DBG] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-25T21:12:04.455 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.453+0000 7f2bbcc40700 1 -- 172.21.15.174:0/727057859 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bb80f5410 msgr2=0x7f2bb80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:04.455 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.453+0000 7f2bbcc40700 1 --2- 172.21.15.174:0/727057859 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bb80f5410 0x7f2bb80f5830 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f2ba8009fb0 tx=0x7f2ba8009b30).stop 2023-12-25T21:12:04.455 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.453+0000 7f2bbcc40700 1 -- 172.21.15.174:0/727057859 shutdown_connections 2023-12-25T21:12:04.455 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.453+0000 7f2bbcc40700 1 --2- 172.21.15.174:0/727057859 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bb80f5410 0x7f2bb80f5830 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:04.456 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.453+0000 7f2bbcc40700 1 -- 172.21.15.174:0/727057859 >> 172.21.15.174:0/727057859 conn(0x7f2bb80f0b70 msgr2=0x7f2bb80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:04.456 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.453+0000 7f2bbcc40700 1 -- 172.21.15.174:0/727057859 shutdown_connections 2023-12-25T21:12:04.456 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.453+0000 7f2bbcc40700 1 -- 172.21.15.174:0/727057859 wait complete. 2023-12-25T21:12:04.456 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.454+0000 7f2bbcc40700 1 Processor -- start 2023-12-25T21:12:04.456 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.454+0000 7f2bbcc40700 1 -- start start 2023-12-25T21:12:04.456 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.454+0000 7f2bbcc40700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bb80f5410 0x7f2bb8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:04.456 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.454+0000 7f2bbcc40700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb8060240 con 0x7f2bb80f5410 2023-12-25T21:12:04.456 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.454+0000 7f2bb659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bb80f5410 0x7f2bb8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:04.456 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.454+0000 7f2bb659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bb80f5410 0x7f2bb8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:53200/0 (socket says 172.21.15.174:53200) 2023-12-25T21:12:04.456 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.454+0000 7f2bb659c700 1 -- 172.21.15.174:0/532732323 learned_addr learned my addr 172.21.15.174:0/532732323 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:04.456 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.454+0000 7f2bb659c700 1 -- 172.21.15.174:0/532732323 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba8005040 con 0x7f2bb80f5410 2023-12-25T21:12:04.456 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.454+0000 7f2bb659c700 1 --2- 172.21.15.174:0/532732323 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bb80f5410 0x7f2bb8109430 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f2ba8013510 tx=0x7f2ba800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:04.456 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.454+0000 7f2baeffd700 1 -- 172.21.15.174:0/532732323 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ba8009320 con 0x7f2bb80f5410 2023-12-25T21:12:04.457 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.454+0000 7f2baeffd700 1 -- 172.21.15.174:0/532732323 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ba8013f00 con 0x7f2bb80f5410 2023-12-25T21:12:04.457 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.454+0000 7f2bbcc40700 1 -- 172.21.15.174:0/532732323 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb8060440 con 0x7f2bb80f5410 2023-12-25T21:12:04.457 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.454+0000 7f2baeffd700 1 -- 172.21.15.174:0/532732323 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ba8014d90 con 0x7f2bb80f5410 2023-12-25T21:12:04.457 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.455+0000 7f2bbcc40700 1 -- 172.21.15.174:0/532732323 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb8109ae0 con 0x7f2bb80f5410 2023-12-25T21:12:04.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.455+0000 7f2baeffd700 1 -- 172.21.15.174:0/532732323 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2ba8014420 con 0x7f2bb80f5410 2023-12-25T21:12:04.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.455+0000 7f2baeffd700 1 --2- 172.21.15.174:0/532732323 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2ba402ff60 0x7f2ba4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:04.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.455+0000 7f2baeffd700 1 -- 172.21.15.174:0/532732323 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2ba80410c0 con 0x7f2bb80f5410 2023-12-25T21:12:04.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.455+0000 7f2bbcc40700 1 -- 172.21.15.174:0/532732323 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb8045ad0 con 0x7f2bb80f5410 2023-12-25T21:12:04.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.455+0000 7f2bb5d9b700 1 --2- 172.21.15.174:0/532732323 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2ba402ff60 0x7f2ba4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:04.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.456+0000 7f2bb5d9b700 1 --2- 172.21.15.174:0/532732323 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2ba402ff60 0x7f2ba4032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f2ba00096a0 tx=0x7f2ba0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:04.461 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.460+0000 7f2baeffd700 1 -- 172.21.15.174:0/532732323 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ba80095f0 con 0x7f2bb80f5410 2023-12-25T21:12:04.668 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.668+0000 7f2bbcc40700 1 -- 172.21.15.174:0/532732323 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2bb80faf50 con 0x7f2bb80f5410 2023-12-25T21:12:04.669 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.668+0000 7f2baeffd700 1 -- 172.21.15.174:0/532732323 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2ba801db70 con 0x7f2bb80f5410 2023-12-25T21:12:04.669 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:04.669 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:04.671 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.670+0000 7f2bbcc40700 1 -- 172.21.15.174:0/532732323 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2ba402ff60 msgr2=0x7f2ba4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:04.671 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.670+0000 7f2bbcc40700 1 --2- 172.21.15.174:0/532732323 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2ba402ff60 0x7f2ba4032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f2ba00096a0 tx=0x7f2ba0006b40).stop 2023-12-25T21:12:04.671 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.670+0000 7f2bbcc40700 1 -- 172.21.15.174:0/532732323 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bb80f5410 msgr2=0x7f2bb8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:04.671 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.670+0000 7f2bbcc40700 1 --2- 172.21.15.174:0/532732323 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bb80f5410 0x7f2bb8109430 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f2ba8013510 tx=0x7f2ba800b040).stop 2023-12-25T21:12:04.671 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.670+0000 7f2bbcc40700 1 -- 172.21.15.174:0/532732323 shutdown_connections 2023-12-25T21:12:04.671 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.670+0000 7f2bbcc40700 1 --2- 172.21.15.174:0/532732323 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f2ba402ff60 0x7f2ba4032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:04.671 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.670+0000 7f2bbcc40700 1 --2- 172.21.15.174:0/532732323 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2bb80f5410 0x7f2bb8109430 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:04.671 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.670+0000 7f2bbcc40700 1 -- 172.21.15.174:0/532732323 >> 172.21.15.174:0/532732323 conn(0x7f2bb80f0b70 msgr2=0x7f2bb80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:04.672 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.670+0000 7f2bbcc40700 1 -- 172.21.15.174:0/532732323 shutdown_connections 2023-12-25T21:12:04.672 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:04.670+0000 7f2bbcc40700 1 -- 172.21.15.174:0/532732323 wait complete. 2023-12-25T21:12:04.672 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:04.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:04 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cephadm 2023-12-25T21:12:02.370063+0000 mgr.smithi061.jxxeke (mgr.14164) 157 : cephadm [INF] Deploying daemon crash.smithi061 on smithi061 2023-12-25T21:12:04.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:04 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:02.615683+0000 mgr.smithi061.jxxeke (mgr.14164) 158 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:04.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:04 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:03.878288+0000 mon.smithi061 (mon.0) 232 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' 2023-12-25T21:12:04.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:04 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:03.978855+0000 mon.smithi061 (mon.0) 233 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' 2023-12-25T21:12:04.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:04 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:03.982047+0000 mon.smithi061 (mon.0) 234 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' 2023-12-25T21:12:04.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:04 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:03.982500+0000 mon.smithi061 (mon.0) 235 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-25T21:12:04.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:04 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:03.985898+0000 mon.smithi061 (mon.0) 236 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' 2023-12-25T21:12:05.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:05 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:03.982908+0000 mgr.smithi061.jxxeke (mgr.14164) 159 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-25T21:12:05.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:05 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cephadm 2023-12-25T21:12:03.988997+0000 mgr.smithi061.jxxeke (mgr.14164) 160 : cephadm [INF] Deploying daemon grafana.smithi061 on smithi061 2023-12-25T21:12:05.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:05 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:04.669749+0000 mon.smithi061 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.174:0/532732323' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:06.065 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:06.065 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:06.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:06 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:04.616031+0000 mgr.smithi061.jxxeke (mgr.14164) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:07.318 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.316+0000 7fe91bf15700 1 -- 172.21.15.174:0/685611705 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9140f6510 msgr2=0x7fe9140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:07.318 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.316+0000 7fe91bf15700 1 --2- 172.21.15.174:0/685611705 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9140f6510 0x7fe9140f6930 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fe908009fb0 tx=0x7fe908009b30).stop 2023-12-25T21:12:07.319 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.316+0000 7fe91bf15700 1 -- 172.21.15.174:0/685611705 shutdown_connections 2023-12-25T21:12:07.319 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.316+0000 7fe91bf15700 1 --2- 172.21.15.174:0/685611705 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9140f6510 0x7fe9140f6930 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:07.319 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.316+0000 7fe91bf15700 1 -- 172.21.15.174:0/685611705 >> 172.21.15.174:0/685611705 conn(0x7fe9140f1c70 msgr2=0x7fe9140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:07.319 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.316+0000 7fe91bf15700 1 -- 172.21.15.174:0/685611705 shutdown_connections 2023-12-25T21:12:07.319 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.317+0000 7fe91bf15700 1 -- 172.21.15.174:0/685611705 wait complete. 2023-12-25T21:12:07.320 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.317+0000 7fe91bf15700 1 Processor -- start 2023-12-25T21:12:07.320 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.317+0000 7fe91bf15700 1 -- start start 2023-12-25T21:12:07.320 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.317+0000 7fe91bf15700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9140f6510 0x7fe914106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:07.320 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.317+0000 7fe91bf15700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe914106c20 con 0x7fe9140f6510 2023-12-25T21:12:07.320 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.318+0000 7fe919cb1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9140f6510 0x7fe914106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:07.320 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.318+0000 7fe919cb1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9140f6510 0x7fe914106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:37650/0 (socket says 172.21.15.174:37650) 2023-12-25T21:12:07.320 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.318+0000 7fe919cb1700 1 -- 172.21.15.174:0/1025487634 learned_addr learned my addr 172.21.15.174:0/1025487634 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:07.320 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.318+0000 7fe919cb1700 1 -- 172.21.15.174:0/1025487634 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe908005040 con 0x7fe9140f6510 2023-12-25T21:12:07.320 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.318+0000 7fe919cb1700 1 --2- 172.21.15.174:0/1025487634 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9140f6510 0x7fe914106730 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fe908013510 tx=0x7fe90800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:07.321 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.318+0000 7fe9067fc700 1 -- 172.21.15.174:0/1025487634 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe908009320 con 0x7fe9140f6510 2023-12-25T21:12:07.321 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.318+0000 7fe91bf15700 1 -- 172.21.15.174:0/1025487634 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe914106e20 con 0x7fe9140f6510 2023-12-25T21:12:07.322 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.318+0000 7fe9067fc700 1 -- 172.21.15.174:0/1025487634 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe908013f00 con 0x7fe9140f6510 2023-12-25T21:12:07.322 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.318+0000 7fe9067fc700 1 -- 172.21.15.174:0/1025487634 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe908014d90 con 0x7fe9140f6510 2023-12-25T21:12:07.322 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.318+0000 7fe91bf15700 1 -- 172.21.15.174:0/1025487634 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9141072b0 con 0x7fe9140f6510 2023-12-25T21:12:07.322 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.319+0000 7fe9067fc700 1 -- 172.21.15.174:0/1025487634 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9080095f0 con 0x7fe9140f6510 2023-12-25T21:12:07.322 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.319+0000 7fe91bf15700 1 -- 172.21.15.174:0/1025487634 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe914045ad0 con 0x7fe9140f6510 2023-12-25T21:12:07.322 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.319+0000 7fe9067fc700 1 --2- 172.21.15.174:0/1025487634 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe90002ff60 0x7fe900032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:07.322 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.319+0000 7fe9067fc700 1 -- 172.21.15.174:0/1025487634 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe908041000 con 0x7fe9140f6510 2023-12-25T21:12:07.322 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.319+0000 7fe9194b0700 1 --2- 172.21.15.174:0/1025487634 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe90002ff60 0x7fe900032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:07.322 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.320+0000 7fe9194b0700 1 --2- 172.21.15.174:0/1025487634 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe90002ff60 0x7fe900032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe9100096a0 tx=0x7fe910006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:07.325 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.324+0000 7fe9067fc700 1 -- 172.21.15.174:0/1025487634 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe908014420 con 0x7fe9140f6510 2023-12-25T21:12:07.529 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.529+0000 7fe91bf15700 1 -- 172.21.15.174:0/1025487634 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9140fbf60 con 0x7fe9140f6510 2023-12-25T21:12:07.530 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.529+0000 7fe9067fc700 1 -- 172.21.15.174:0/1025487634 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe90801d7f0 con 0x7fe9140f6510 2023-12-25T21:12:07.530 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:07.530 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:07.532 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.531+0000 7fe91bf15700 1 -- 172.21.15.174:0/1025487634 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe90002ff60 msgr2=0x7fe900032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:07.532 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.531+0000 7fe91bf15700 1 --2- 172.21.15.174:0/1025487634 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe90002ff60 0x7fe900032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe9100096a0 tx=0x7fe910006b40).stop 2023-12-25T21:12:07.532 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.531+0000 7fe91bf15700 1 -- 172.21.15.174:0/1025487634 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9140f6510 msgr2=0x7fe914106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:07.532 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.531+0000 7fe91bf15700 1 --2- 172.21.15.174:0/1025487634 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9140f6510 0x7fe914106730 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fe908013510 tx=0x7fe90800b040).stop 2023-12-25T21:12:07.532 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.531+0000 7fe91bf15700 1 -- 172.21.15.174:0/1025487634 shutdown_connections 2023-12-25T21:12:07.532 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.531+0000 7fe91bf15700 1 --2- 172.21.15.174:0/1025487634 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe90002ff60 0x7fe900032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:07.532 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.531+0000 7fe91bf15700 1 --2- 172.21.15.174:0/1025487634 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9140f6510 0x7fe914106730 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:07.532 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.531+0000 7fe91bf15700 1 -- 172.21.15.174:0/1025487634 >> 172.21.15.174:0/1025487634 conn(0x7fe9140f1c70 msgr2=0x7fe9140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:07.532 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.531+0000 7fe91bf15700 1 -- 172.21.15.174:0/1025487634 shutdown_connections 2023-12-25T21:12:07.532 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:07.531+0000 7fe91bf15700 1 -- 172.21.15.174:0/1025487634 wait complete. 2023-12-25T21:12:07.533 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:08.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:08 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:06.616298+0000 mgr.smithi061.jxxeke (mgr.14164) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:08.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:08 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:07.530721+0000 mon.smithi061 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.174:0/1025487634' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:08.859 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:08.860 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:10.017 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.015+0000 7f5cd61b9700 1 -- 172.21.15.174:0/3439763036 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd00f5410 msgr2=0x7f5cd00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:10.017 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.015+0000 7f5cd61b9700 1 --2- 172.21.15.174:0/3439763036 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd00f5410 0x7f5cd00f5830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f5cc4009fb0 tx=0x7f5cc4009b30).stop 2023-12-25T21:12:10.017 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.015+0000 7f5cd61b9700 1 -- 172.21.15.174:0/3439763036 shutdown_connections 2023-12-25T21:12:10.017 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.015+0000 7f5cd61b9700 1 --2- 172.21.15.174:0/3439763036 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd00f5410 0x7f5cd00f5830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:10.017 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.015+0000 7f5cd61b9700 1 -- 172.21.15.174:0/3439763036 >> 172.21.15.174:0/3439763036 conn(0x7f5cd00f0b70 msgr2=0x7f5cd00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:10.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.015+0000 7f5cd61b9700 1 -- 172.21.15.174:0/3439763036 shutdown_connections 2023-12-25T21:12:10.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.015+0000 7f5cd61b9700 1 -- 172.21.15.174:0/3439763036 wait complete. 2023-12-25T21:12:10.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.016+0000 7f5cd61b9700 1 Processor -- start 2023-12-25T21:12:10.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.016+0000 7f5cd61b9700 1 -- start start 2023-12-25T21:12:10.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.016+0000 7f5cd61b9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd00f5410 0x7f5cd0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:10.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.016+0000 7f5cd61b9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd0105b20 con 0x7f5cd00f5410 2023-12-25T21:12:10.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.016+0000 7f5ccf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd00f5410 0x7f5cd0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:10.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.017+0000 7f5ccf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd00f5410 0x7f5cd0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:37656/0 (socket says 172.21.15.174:37656) 2023-12-25T21:12:10.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.017+0000 7f5ccf7fe700 1 -- 172.21.15.174:0/51715309 learned_addr learned my addr 172.21.15.174:0/51715309 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:10.019 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.017+0000 7f5ccf7fe700 1 -- 172.21.15.174:0/51715309 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc4005040 con 0x7f5cd00f5410 2023-12-25T21:12:10.020 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.017+0000 7f5ccf7fe700 1 --2- 172.21.15.174:0/51715309 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd00f5410 0x7f5cd0105630 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f5cc4014000 tx=0x7f5cc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:10.020 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.017+0000 7f5cb7fff700 1 -- 172.21.15.174:0/51715309 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cc40095f0 con 0x7f5cd00f5410 2023-12-25T21:12:10.020 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.017+0000 7f5cd61b9700 1 -- 172.21.15.174:0/51715309 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cd0105d20 con 0x7f5cd00f5410 2023-12-25T21:12:10.020 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.017+0000 7f5cb7fff700 1 -- 172.21.15.174:0/51715309 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5cc4007e70 con 0x7f5cd00f5410 2023-12-25T21:12:10.020 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.017+0000 7f5cb7fff700 1 -- 172.21.15.174:0/51715309 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cc4014ea0 con 0x7f5cd00f5410 2023-12-25T21:12:10.020 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.017+0000 7f5cd61b9700 1 -- 172.21.15.174:0/51715309 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cd01061b0 con 0x7f5cd00f5410 2023-12-25T21:12:10.020 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.018+0000 7f5cb7fff700 1 -- 172.21.15.174:0/51715309 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5cc4025070 con 0x7f5cd00f5410 2023-12-25T21:12:10.020 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.018+0000 7f5cb7fff700 1 --2- 172.21.15.174:0/51715309 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5cb802ff60 0x7f5cb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:10.020 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.018+0000 7f5cb7fff700 1 -- 172.21.15.174:0/51715309 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5cc4040ab0 con 0x7f5cd00f5410 2023-12-25T21:12:10.020 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.018+0000 7f5cd61b9700 1 -- 172.21.15.174:0/51715309 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cd0045ad0 con 0x7f5cd00f5410 2023-12-25T21:12:10.020 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.018+0000 7f5cceffd700 1 --2- 172.21.15.174:0/51715309 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5cb802ff60 0x7f5cb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:10.020 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.019+0000 7f5cceffd700 1 --2- 172.21.15.174:0/51715309 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5cb802ff60 0x7f5cb8032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f5cc00096a0 tx=0x7f5cc0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:10.024 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.023+0000 7f5cb7fff700 1 -- 172.21.15.174:0/51715309 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5cc4026350 con 0x7f5cd00f5410 2023-12-25T21:12:10.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.230+0000 7f5cd61b9700 1 -- 172.21.15.174:0/51715309 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5cd0106ef0 con 0x7f5cd00f5410 2023-12-25T21:12:10.232 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.231+0000 7f5cb7fff700 1 -- 172.21.15.174:0/51715309 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5cc4014690 con 0x7f5cd00f5410 2023-12-25T21:12:10.232 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:10.232 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:10.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.233+0000 7f5cd61b9700 1 -- 172.21.15.174:0/51715309 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5cb802ff60 msgr2=0x7f5cb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:10.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.233+0000 7f5cd61b9700 1 --2- 172.21.15.174:0/51715309 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5cb802ff60 0x7f5cb8032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f5cc00096a0 tx=0x7f5cc0006b40).stop 2023-12-25T21:12:10.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.233+0000 7f5cd61b9700 1 -- 172.21.15.174:0/51715309 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd00f5410 msgr2=0x7f5cd0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:10.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.233+0000 7f5cd61b9700 1 --2- 172.21.15.174:0/51715309 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd00f5410 0x7f5cd0105630 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f5cc4014000 tx=0x7f5cc400b040).stop 2023-12-25T21:12:10.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.233+0000 7f5cd61b9700 1 -- 172.21.15.174:0/51715309 shutdown_connections 2023-12-25T21:12:10.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.233+0000 7f5cd61b9700 1 --2- 172.21.15.174:0/51715309 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5cb802ff60 0x7f5cb8032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:10.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.233+0000 7f5cd61b9700 1 --2- 172.21.15.174:0/51715309 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd00f5410 0x7f5cd0105630 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:10.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.233+0000 7f5cd61b9700 1 -- 172.21.15.174:0/51715309 >> 172.21.15.174:0/51715309 conn(0x7f5cd00f0b70 msgr2=0x7f5cd00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:10.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.233+0000 7f5cd61b9700 1 -- 172.21.15.174:0/51715309 shutdown_connections 2023-12-25T21:12:10.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:10.233+0000 7f5cd61b9700 1 -- 172.21.15.174:0/51715309 wait complete. 2023-12-25T21:12:10.235 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:10.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:10 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:08.616513+0000 mgr.smithi061.jxxeke (mgr.14164) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:10.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:10 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:10.232603+0000 mon.smithi061 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.174:0/51715309' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:11.649 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:11.649 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:12.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:12 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:10.616776+0000 mgr.smithi061.jxxeke (mgr.14164) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:12.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.776+0000 7f3ac6471700 1 -- 172.21.15.174:0/4055784886 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ac0060280 msgr2=0x7f3ac00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:12.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.776+0000 7f3ac6471700 1 --2- 172.21.15.174:0/4055784886 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ac0060280 0x7f3ac00f90f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f3ab4009fb0 tx=0x7f3ab4009b30).stop 2023-12-25T21:12:12.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.776+0000 7f3ac6471700 1 -- 172.21.15.174:0/4055784886 shutdown_connections 2023-12-25T21:12:12.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.776+0000 7f3ac6471700 1 --2- 172.21.15.174:0/4055784886 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ac0060280 0x7f3ac00f90f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:12.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.776+0000 7f3ac6471700 1 -- 172.21.15.174:0/4055784886 >> 172.21.15.174:0/4055784886 conn(0x7f3ac00f0b70 msgr2=0x7f3ac00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:12.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.776+0000 7f3ac6471700 1 -- 172.21.15.174:0/4055784886 shutdown_connections 2023-12-25T21:12:12.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.776+0000 7f3ac6471700 1 -- 172.21.15.174:0/4055784886 wait complete. 2023-12-25T21:12:12.781 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.776+0000 7f3ac6471700 1 Processor -- start 2023-12-25T21:12:12.781 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.776+0000 7f3ac6471700 1 -- start start 2023-12-25T21:12:12.781 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.777+0000 7f3ac6471700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ac0060280 0x7f3ac01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:12.781 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.777+0000 7f3ac6471700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac0106bd0 con 0x7f3ac0060280 2023-12-25T21:12:12.781 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.777+0000 7f3abffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ac0060280 0x7f3ac01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:12.781 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.777+0000 7f3abffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ac0060280 0x7f3ac01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:37676/0 (socket says 172.21.15.174:37676) 2023-12-25T21:12:12.781 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.777+0000 7f3abffff700 1 -- 172.21.15.174:0/2589213034 learned_addr learned my addr 172.21.15.174:0/2589213034 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:12.781 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.777+0000 7f3abffff700 1 -- 172.21.15.174:0/2589213034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab4005040 con 0x7f3ac0060280 2023-12-25T21:12:12.782 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.777+0000 7f3abffff700 1 --2- 172.21.15.174:0/2589213034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ac0060280 0x7f3ac01066e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f3ab4013510 tx=0x7f3ab400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:12.782 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.777+0000 7f3abcff9700 1 -- 172.21.15.174:0/2589213034 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ab4009320 con 0x7f3ac0060280 2023-12-25T21:12:12.782 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.777+0000 7f3abcff9700 1 -- 172.21.15.174:0/2589213034 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ab4013f00 con 0x7f3ac0060280 2023-12-25T21:12:12.782 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.777+0000 7f3abcff9700 1 -- 172.21.15.174:0/2589213034 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ab4014dc0 con 0x7f3ac0060280 2023-12-25T21:12:12.782 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.777+0000 7f3ac6471700 1 -- 172.21.15.174:0/2589213034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac0106dd0 con 0x7f3ac0060280 2023-12-25T21:12:12.782 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.777+0000 7f3ac6471700 1 -- 172.21.15.174:0/2589213034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac0107260 con 0x7f3ac0060280 2023-12-25T21:12:12.782 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.778+0000 7f3abcff9700 1 -- 172.21.15.174:0/2589213034 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ab4014420 con 0x7f3ac0060280 2023-12-25T21:12:12.782 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.778+0000 7f3ac6471700 1 -- 172.21.15.174:0/2589213034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac0045ad0 con 0x7f3ac0060280 2023-12-25T21:12:12.783 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.778+0000 7f3abcff9700 1 --2- 172.21.15.174:0/2589213034 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3aa802ffb0 0x7f3aa8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:12.783 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.778+0000 7f3abcff9700 1 -- 172.21.15.174:0/2589213034 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ab40410b0 con 0x7f3ac0060280 2023-12-25T21:12:12.783 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.778+0000 7f3abf7fe700 1 --2- 172.21.15.174:0/2589213034 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3aa802ffb0 0x7f3aa8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:12.783 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.779+0000 7f3abf7fe700 1 --2- 172.21.15.174:0/2589213034 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3aa802ffb0 0x7f3aa8032470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f3ab00096a0 tx=0x7f3ab0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:12.784 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.783+0000 7f3abcff9700 1 -- 172.21.15.174:0/2589213034 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ab406d2b0 con 0x7f3ac0060280 2023-12-25T21:12:12.988 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.987+0000 7f3ac6471700 1 -- 172.21.15.174:0/2589213034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ac0108110 con 0x7f3ac0060280 2023-12-25T21:12:12.989 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.988+0000 7f3abcff9700 1 -- 172.21.15.174:0/2589213034 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3ab40095f0 con 0x7f3ac0060280 2023-12-25T21:12:12.989 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:12.989 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:12.990 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.989+0000 7f3ac6471700 1 -- 172.21.15.174:0/2589213034 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3aa802ffb0 msgr2=0x7f3aa8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:12.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.989+0000 7f3ac6471700 1 --2- 172.21.15.174:0/2589213034 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3aa802ffb0 0x7f3aa8032470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f3ab00096a0 tx=0x7f3ab0006b40).stop 2023-12-25T21:12:12.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.989+0000 7f3ac6471700 1 -- 172.21.15.174:0/2589213034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ac0060280 msgr2=0x7f3ac01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:12.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.989+0000 7f3ac6471700 1 --2- 172.21.15.174:0/2589213034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ac0060280 0x7f3ac01066e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f3ab4013510 tx=0x7f3ab400b040).stop 2023-12-25T21:12:12.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.989+0000 7f3ac6471700 1 -- 172.21.15.174:0/2589213034 shutdown_connections 2023-12-25T21:12:12.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.989+0000 7f3ac6471700 1 --2- 172.21.15.174:0/2589213034 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3aa802ffb0 0x7f3aa8032470 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:12.991 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.989+0000 7f3ac6471700 1 --2- 172.21.15.174:0/2589213034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ac0060280 0x7f3ac01066e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:12.992 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.989+0000 7f3ac6471700 1 -- 172.21.15.174:0/2589213034 >> 172.21.15.174:0/2589213034 conn(0x7f3ac00f0b70 msgr2=0x7f3ac00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:12.992 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.989+0000 7f3ac6471700 1 -- 172.21.15.174:0/2589213034 shutdown_connections 2023-12-25T21:12:12.992 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:12.989+0000 7f3ac6471700 1 -- 172.21.15.174:0/2589213034 wait complete. 2023-12-25T21:12:12.992 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:13.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:13 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:12.989038+0000 mon.smithi061 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.174:0/2589213034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:14.382 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:14.382 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:14.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:14 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:12.617025+0000 mgr.smithi061.jxxeke (mgr.14164) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:15.537 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.534+0000 7fed35fae700 1 -- 172.21.15.174:0/2018888806 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed30060280 msgr2=0x7fed300fb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:15.537 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.534+0000 7fed35fae700 1 --2- 172.21.15.174:0/2018888806 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed30060280 0x7fed300fb4f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fed20009fb0 tx=0x7fed20009b30).stop 2023-12-25T21:12:15.537 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.534+0000 7fed35fae700 1 -- 172.21.15.174:0/2018888806 shutdown_connections 2023-12-25T21:12:15.537 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.534+0000 7fed35fae700 1 --2- 172.21.15.174:0/2018888806 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed30060280 0x7fed300fb4f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:15.537 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.534+0000 7fed35fae700 1 -- 172.21.15.174:0/2018888806 >> 172.21.15.174:0/2018888806 conn(0x7fed300f0b70 msgr2=0x7fed300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:15.537 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.534+0000 7fed35fae700 1 -- 172.21.15.174:0/2018888806 shutdown_connections 2023-12-25T21:12:15.537 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.534+0000 7fed35fae700 1 -- 172.21.15.174:0/2018888806 wait complete. 2023-12-25T21:12:15.537 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.535+0000 7fed35fae700 1 Processor -- start 2023-12-25T21:12:15.537 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.535+0000 7fed35fae700 1 -- start start 2023-12-25T21:12:15.537 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.535+0000 7fed35fae700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed30060280 0x7fed301088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:15.537 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.535+0000 7fed35fae700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed30108dd0 con 0x7fed30060280 2023-12-25T21:12:15.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.535+0000 7fed2f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed30060280 0x7fed301088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:15.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.535+0000 7fed2f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed30060280 0x7fed301088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:37694/0 (socket says 172.21.15.174:37694) 2023-12-25T21:12:15.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.535+0000 7fed2f7fe700 1 -- 172.21.15.174:0/2028286596 learned_addr learned my addr 172.21.15.174:0/2028286596 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:15.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.536+0000 7fed2f7fe700 1 -- 172.21.15.174:0/2028286596 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed20005040 con 0x7fed30060280 2023-12-25T21:12:15.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.536+0000 7fed2f7fe700 1 --2- 172.21.15.174:0/2028286596 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed30060280 0x7fed301088e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fed20009830 tx=0x7fed2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:15.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.536+0000 7fed27fff700 1 -- 172.21.15.174:0/2028286596 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed20009320 con 0x7fed30060280 2023-12-25T21:12:15.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.536+0000 7fed35fae700 1 -- 172.21.15.174:0/2028286596 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed30108fd0 con 0x7fed30060280 2023-12-25T21:12:15.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.536+0000 7fed27fff700 1 -- 172.21.15.174:0/2028286596 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed200136c0 con 0x7fed30060280 2023-12-25T21:12:15.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.536+0000 7fed27fff700 1 -- 172.21.15.174:0/2028286596 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed2001ee90 con 0x7fed30060280 2023-12-25T21:12:15.538 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.536+0000 7fed35fae700 1 -- 172.21.15.174:0/2028286596 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed30109460 con 0x7fed30060280 2023-12-25T21:12:15.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.537+0000 7fed27fff700 1 -- 172.21.15.174:0/2028286596 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed2001e420 con 0x7fed30060280 2023-12-25T21:12:15.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.537+0000 7fed27fff700 1 --2- 172.21.15.174:0/2028286596 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fed1c02ff10 0x7fed1c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:15.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.537+0000 7fed27fff700 1 -- 172.21.15.174:0/2028286596 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed20040fe0 con 0x7fed30060280 2023-12-25T21:12:15.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.537+0000 7fed35fae700 1 -- 172.21.15.174:0/2028286596 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed30045ad0 con 0x7fed30060280 2023-12-25T21:12:15.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.537+0000 7fed2effd700 1 --2- 172.21.15.174:0/2028286596 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fed1c02ff10 0x7fed1c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:15.539 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.537+0000 7fed2effd700 1 --2- 172.21.15.174:0/2028286596 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fed1c02ff10 0x7fed1c0323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fed180096a0 tx=0x7fed18006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:15.542 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.542+0000 7fed27fff700 1 -- 172.21.15.174:0/2028286596 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed2001cb50 con 0x7fed30060280 2023-12-25T21:12:15.749 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.748+0000 7fed35fae700 1 -- 172.21.15.174:0/2028286596 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed3010a220 con 0x7fed30060280 2023-12-25T21:12:15.749 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.748+0000 7fed27fff700 1 -- 172.21.15.174:0/2028286596 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fed2001c320 con 0x7fed30060280 2023-12-25T21:12:15.750 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:15.750 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:15.751 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.750+0000 7fed35fae700 1 -- 172.21.15.174:0/2028286596 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fed1c02ff10 msgr2=0x7fed1c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:15.751 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.750+0000 7fed35fae700 1 --2- 172.21.15.174:0/2028286596 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fed1c02ff10 0x7fed1c0323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fed180096a0 tx=0x7fed18006b40).stop 2023-12-25T21:12:15.751 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.750+0000 7fed35fae700 1 -- 172.21.15.174:0/2028286596 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed30060280 msgr2=0x7fed301088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:15.751 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.750+0000 7fed35fae700 1 --2- 172.21.15.174:0/2028286596 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed30060280 0x7fed301088e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fed20009830 tx=0x7fed2000b040).stop 2023-12-25T21:12:15.751 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.750+0000 7fed35fae700 1 -- 172.21.15.174:0/2028286596 shutdown_connections 2023-12-25T21:12:15.751 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.750+0000 7fed35fae700 1 --2- 172.21.15.174:0/2028286596 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fed1c02ff10 0x7fed1c0323d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:15.751 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.750+0000 7fed35fae700 1 --2- 172.21.15.174:0/2028286596 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed30060280 0x7fed301088e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:15.751 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.750+0000 7fed35fae700 1 -- 172.21.15.174:0/2028286596 >> 172.21.15.174:0/2028286596 conn(0x7fed300f0b70 msgr2=0x7fed300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:15.752 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.750+0000 7fed35fae700 1 -- 172.21.15.174:0/2028286596 shutdown_connections 2023-12-25T21:12:15.752 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:15.750+0000 7fed35fae700 1 -- 172.21.15.174:0/2028286596 wait complete. 2023-12-25T21:12:15.752 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:16.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:16 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:14.617252+0000 mgr.smithi061.jxxeke (mgr.14164) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:16.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:16 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:15.749754+0000 mon.smithi061 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.174:0/2028286596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:17.152 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:17.153 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:18.374 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.373+0000 7f84b3c49700 1 -- 172.21.15.174:0/3769054844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84ac0f5410 msgr2=0x7f84ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:18.374 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.373+0000 7f84b3c49700 1 --2- 172.21.15.174:0/3769054844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84ac0f5410 0x7f84ac0f5830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f849c009fb0 tx=0x7f849c009b30).stop 2023-12-25T21:12:18.374 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.373+0000 7f84b3c49700 1 -- 172.21.15.174:0/3769054844 shutdown_connections 2023-12-25T21:12:18.375 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.373+0000 7f84b3c49700 1 --2- 172.21.15.174:0/3769054844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84ac0f5410 0x7f84ac0f5830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:18.375 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.373+0000 7f84b3c49700 1 -- 172.21.15.174:0/3769054844 >> 172.21.15.174:0/3769054844 conn(0x7f84ac0f0b70 msgr2=0x7f84ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:18.375 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.373+0000 7f84b3c49700 1 -- 172.21.15.174:0/3769054844 shutdown_connections 2023-12-25T21:12:18.375 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.373+0000 7f84b3c49700 1 -- 172.21.15.174:0/3769054844 wait complete. 2023-12-25T21:12:18.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.374+0000 7f84b3c49700 1 Processor -- start 2023-12-25T21:12:18.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.374+0000 7f84b3c49700 1 -- start start 2023-12-25T21:12:18.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.374+0000 7f84b3c49700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84ac0f5410 0x7f84ac105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:18.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.374+0000 7f84b3c49700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ac105b20 con 0x7f84ac0f5410 2023-12-25T21:12:18.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.375+0000 7f84b19e5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84ac0f5410 0x7f84ac105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:18.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.375+0000 7f84b19e5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84ac0f5410 0x7f84ac105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:53836/0 (socket says 172.21.15.174:53836) 2023-12-25T21:12:18.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.375+0000 7f84b19e5700 1 -- 172.21.15.174:0/2779987844 learned_addr learned my addr 172.21.15.174:0/2779987844 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:18.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.375+0000 7f84b19e5700 1 -- 172.21.15.174:0/2779987844 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849c005040 con 0x7f84ac0f5410 2023-12-25T21:12:18.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.375+0000 7f84b19e5700 1 --2- 172.21.15.174:0/2779987844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84ac0f5410 0x7f84ac105630 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f849c013510 tx=0x7f849c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:18.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.375+0000 7f84a27fc700 1 -- 172.21.15.174:0/2779987844 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f849c009320 con 0x7f84ac0f5410 2023-12-25T21:12:18.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.375+0000 7f84a27fc700 1 -- 172.21.15.174:0/2779987844 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f849c013f00 con 0x7f84ac0f5410 2023-12-25T21:12:18.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.375+0000 7f84a27fc700 1 -- 172.21.15.174:0/2779987844 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f849c014d90 con 0x7f84ac0f5410 2023-12-25T21:12:18.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.375+0000 7f84b3c49700 1 -- 172.21.15.174:0/2779987844 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84ac105d20 con 0x7f84ac0f5410 2023-12-25T21:12:18.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.375+0000 7f84b3c49700 1 -- 172.21.15.174:0/2779987844 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84ac1061b0 con 0x7f84ac0f5410 2023-12-25T21:12:18.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.376+0000 7f84a27fc700 1 -- 172.21.15.174:0/2779987844 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f849c014420 con 0x7f84ac0f5410 2023-12-25T21:12:18.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.376+0000 7f84a27fc700 1 --2- 172.21.15.174:0/2779987844 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f849802ffb0 0x7f8498032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:18.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.376+0000 7f84a27fc700 1 -- 172.21.15.174:0/2779987844 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f849c0410c0 con 0x7f84ac0f5410 2023-12-25T21:12:18.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.376+0000 7f84b3c49700 1 -- 172.21.15.174:0/2779987844 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84ac045ad0 con 0x7f84ac0f5410 2023-12-25T21:12:18.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.376+0000 7f84b11e4700 1 --2- 172.21.15.174:0/2779987844 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f849802ffb0 0x7f8498032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:18.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.377+0000 7f84b11e4700 1 --2- 172.21.15.174:0/2779987844 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f849802ffb0 0x7f8498032470 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f84a80096a0 tx=0x7f84a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:18.381 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.381+0000 7f84a27fc700 1 -- 172.21.15.174:0/2779987844 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f849c0095f0 con 0x7f84ac0f5410 2023-12-25T21:12:18.585 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.584+0000 7f84b3c49700 1 -- 172.21.15.174:0/2779987844 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84ac0fae60 con 0x7f84ac0f5410 2023-12-25T21:12:18.586 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.585+0000 7f84a27fc700 1 -- 172.21.15.174:0/2779987844 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f849c01db70 con 0x7f84ac0f5410 2023-12-25T21:12:18.586 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:18.586 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:18.587 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.586+0000 7f84b3c49700 1 -- 172.21.15.174:0/2779987844 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f849802ffb0 msgr2=0x7f8498032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:18.587 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.586+0000 7f84b3c49700 1 --2- 172.21.15.174:0/2779987844 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f849802ffb0 0x7f8498032470 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f84a80096a0 tx=0x7f84a8006b40).stop 2023-12-25T21:12:18.587 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.586+0000 7f84b3c49700 1 -- 172.21.15.174:0/2779987844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84ac0f5410 msgr2=0x7f84ac105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:18.588 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.586+0000 7f84b3c49700 1 --2- 172.21.15.174:0/2779987844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84ac0f5410 0x7f84ac105630 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f849c013510 tx=0x7f849c00b040).stop 2023-12-25T21:12:18.588 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.586+0000 7f84b3c49700 1 -- 172.21.15.174:0/2779987844 shutdown_connections 2023-12-25T21:12:18.588 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.586+0000 7f84b3c49700 1 --2- 172.21.15.174:0/2779987844 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f849802ffb0 0x7f8498032470 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:18.588 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.586+0000 7f84b3c49700 1 --2- 172.21.15.174:0/2779987844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84ac0f5410 0x7f84ac105630 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:18.588 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.586+0000 7f84b3c49700 1 -- 172.21.15.174:0/2779987844 >> 172.21.15.174:0/2779987844 conn(0x7f84ac0f0b70 msgr2=0x7f84ac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:18.588 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.586+0000 7f84b3c49700 1 -- 172.21.15.174:0/2779987844 shutdown_connections 2023-12-25T21:12:18.588 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:18.587+0000 7f84b3c49700 1 -- 172.21.15.174:0/2779987844 wait complete. 2023-12-25T21:12:18.588 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:18.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:18 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:16.617507+0000 mgr.smithi061.jxxeke (mgr.14164) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:19.681 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:19 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:18.586406+0000 mon.smithi061 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.174:0/2779987844' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:19.991 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:19.992 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:20.455 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:20 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:18.617772+0000 mgr.smithi061.jxxeke (mgr.14164) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:21.159 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.158+0000 7f30a4e84700 1 -- 172.21.15.174:0/2109921742 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a00f51a0 msgr2=0x7f30a00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:21.159 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.158+0000 7f30a4e84700 1 --2- 172.21.15.174:0/2109921742 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a00f51a0 0x7f30a00f55c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f3094009fb0 tx=0x7f3094009b30).stop 2023-12-25T21:12:21.159 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.158+0000 7f30a4e84700 1 -- 172.21.15.174:0/2109921742 shutdown_connections 2023-12-25T21:12:21.159 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.158+0000 7f30a4e84700 1 --2- 172.21.15.174:0/2109921742 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a00f51a0 0x7f30a00f55c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:21.159 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.158+0000 7f30a4e84700 1 -- 172.21.15.174:0/2109921742 >> 172.21.15.174:0/2109921742 conn(0x7f30a00f09a0 msgr2=0x7f30a00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:21.159 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.158+0000 7f30a4e84700 1 -- 172.21.15.174:0/2109921742 shutdown_connections 2023-12-25T21:12:21.159 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.158+0000 7f30a4e84700 1 -- 172.21.15.174:0/2109921742 wait complete. 2023-12-25T21:12:21.162 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.158+0000 7f30a4e84700 1 Processor -- start 2023-12-25T21:12:21.162 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.159+0000 7f30a4e84700 1 -- start start 2023-12-25T21:12:21.162 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.159+0000 7f30a4e84700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a00f51a0 0x7f30a01053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:21.162 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.159+0000 7f30a4e84700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a01058a0 con 0x7f30a00f51a0 2023-12-25T21:12:21.162 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.159+0000 7f309f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a00f51a0 0x7f30a01053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:21.162 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.159+0000 7f309f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a00f51a0 0x7f30a01053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:53854/0 (socket says 172.21.15.174:53854) 2023-12-25T21:12:21.162 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.159+0000 7f309f7fe700 1 -- 172.21.15.174:0/2819049716 learned_addr learned my addr 172.21.15.174:0/2819049716 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:21.162 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.159+0000 7f309f7fe700 1 -- 172.21.15.174:0/2819049716 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3094005040 con 0x7f30a00f51a0 2023-12-25T21:12:21.162 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.159+0000 7f309f7fe700 1 --2- 172.21.15.174:0/2819049716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a00f51a0 0x7f30a01053b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f3094013510 tx=0x7f309400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:21.162 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.160+0000 7f3087fff700 1 -- 172.21.15.174:0/2819049716 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3094009320 con 0x7f30a00f51a0 2023-12-25T21:12:21.163 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.160+0000 7f3087fff700 1 -- 172.21.15.174:0/2819049716 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3094013f00 con 0x7f30a00f51a0 2023-12-25T21:12:21.163 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.160+0000 7f30a4e84700 1 -- 172.21.15.174:0/2819049716 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a0105aa0 con 0x7f30a00f51a0 2023-12-25T21:12:21.163 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.160+0000 7f3087fff700 1 -- 172.21.15.174:0/2819049716 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3094014dc0 con 0x7f30a00f51a0 2023-12-25T21:12:21.163 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.160+0000 7f30a4e84700 1 -- 172.21.15.174:0/2819049716 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a0105f30 con 0x7f30a00f51a0 2023-12-25T21:12:21.163 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.160+0000 7f3087fff700 1 -- 172.21.15.174:0/2819049716 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30940095f0 con 0x7f30a00f51a0 2023-12-25T21:12:21.163 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.160+0000 7f30a4e84700 1 -- 172.21.15.174:0/2819049716 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a0045ad0 con 0x7f30a00f51a0 2023-12-25T21:12:21.163 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.161+0000 7f3087fff700 1 --2- 172.21.15.174:0/2819049716 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f308802ff60 0x7f3088032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:21.163 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.161+0000 7f3087fff700 1 -- 172.21.15.174:0/2819049716 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30940410c0 con 0x7f30a00f51a0 2023-12-25T21:12:21.163 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.161+0000 7f309effd700 1 --2- 172.21.15.174:0/2819049716 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f308802ff60 0x7f3088032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:21.163 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.161+0000 7f309effd700 1 --2- 172.21.15.174:0/2819049716 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f308802ff60 0x7f3088032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f30900096a0 tx=0x7f3090006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:21.167 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.166+0000 7f3087fff700 1 -- 172.21.15.174:0/2819049716 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3094014420 con 0x7f30a00f51a0 2023-12-25T21:12:21.375 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.373+0000 7f30a4e84700 1 -- 172.21.15.174:0/2819049716 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30a0106de0 con 0x7f30a00f51a0 2023-12-25T21:12:21.375 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.373+0000 7f3087fff700 1 -- 172.21.15.174:0/2819049716 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3094010230 con 0x7f30a00f51a0 2023-12-25T21:12:21.375 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:21.375 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:21.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.375+0000 7f30a4e84700 1 -- 172.21.15.174:0/2819049716 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f308802ff60 msgr2=0x7f3088032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:21.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.375+0000 7f30a4e84700 1 --2- 172.21.15.174:0/2819049716 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f308802ff60 0x7f3088032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f30900096a0 tx=0x7f3090006b40).stop 2023-12-25T21:12:21.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.375+0000 7f30a4e84700 1 -- 172.21.15.174:0/2819049716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a00f51a0 msgr2=0x7f30a01053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:21.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.375+0000 7f30a4e84700 1 --2- 172.21.15.174:0/2819049716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a00f51a0 0x7f30a01053b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f3094013510 tx=0x7f309400b040).stop 2023-12-25T21:12:21.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.375+0000 7f30a4e84700 1 -- 172.21.15.174:0/2819049716 shutdown_connections 2023-12-25T21:12:21.377 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.375+0000 7f30a4e84700 1 --2- 172.21.15.174:0/2819049716 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f308802ff60 0x7f3088032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:21.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.375+0000 7f30a4e84700 1 --2- 172.21.15.174:0/2819049716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a00f51a0 0x7f30a01053b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:21.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.375+0000 7f30a4e84700 1 -- 172.21.15.174:0/2819049716 >> 172.21.15.174:0/2819049716 conn(0x7f30a00f09a0 msgr2=0x7f30a00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:21.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.375+0000 7f30a4e84700 1 -- 172.21.15.174:0/2819049716 shutdown_connections 2023-12-25T21:12:21.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:21.375+0000 7f30a4e84700 1 -- 172.21.15.174:0/2819049716 wait complete. 2023-12-25T21:12:21.378 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:22.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:22 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:20.618156+0000 mgr.smithi061.jxxeke (mgr.14164) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:22.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:22 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:21.375076+0000 mon.smithi061 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.174:0/2819049716' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:22.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:22 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:21.831945+0000 mon.smithi061 (mon.0) 244 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' 2023-12-25T21:12:22.763 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:22.763 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:23.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cephadm 2023-12-25T21:12:21.833358+0000 mgr.smithi061.jxxeke (mgr.14164) 170 : cephadm [INF] Deploying daemon node-exporter.smithi061 on smithi061 2023-12-25T21:12:23.993 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.991+0000 7fd519155700 1 -- 172.21.15.174:0/231836646 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5140f5410 msgr2=0x7fd5140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:23.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.991+0000 7fd519155700 1 --2- 172.21.15.174:0/231836646 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5140f5410 0x7fd5140f5830 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fd4fc009fb0 tx=0x7fd4fc009b30).stop 2023-12-25T21:12:23.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.991+0000 7fd519155700 1 -- 172.21.15.174:0/231836646 shutdown_connections 2023-12-25T21:12:23.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.991+0000 7fd519155700 1 --2- 172.21.15.174:0/231836646 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5140f5410 0x7fd5140f5830 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:23.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.991+0000 7fd519155700 1 -- 172.21.15.174:0/231836646 >> 172.21.15.174:0/231836646 conn(0x7fd5140f0b70 msgr2=0x7fd5140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:23.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.991+0000 7fd519155700 1 -- 172.21.15.174:0/231836646 shutdown_connections 2023-12-25T21:12:23.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.991+0000 7fd519155700 1 -- 172.21.15.174:0/231836646 wait complete. 2023-12-25T21:12:23.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.992+0000 7fd519155700 1 Processor -- start 2023-12-25T21:12:23.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.992+0000 7fd519155700 1 -- start start 2023-12-25T21:12:23.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.992+0000 7fd519155700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5140f5410 0x7fd514105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:23.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.992+0000 7fd519155700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd514105b20 con 0x7fd5140f5410 2023-12-25T21:12:23.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.992+0000 7fd512d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5140f5410 0x7fd514105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:23.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.992+0000 7fd512d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5140f5410 0x7fd514105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:53874/0 (socket says 172.21.15.174:53874) 2023-12-25T21:12:23.995 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.992+0000 7fd512d9d700 1 -- 172.21.15.174:0/137039372 learned_addr learned my addr 172.21.15.174:0/137039372 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:23.995 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.993+0000 7fd512d9d700 1 -- 172.21.15.174:0/137039372 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4fc005040 con 0x7fd5140f5410 2023-12-25T21:12:23.995 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.993+0000 7fd512d9d700 1 --2- 172.21.15.174:0/137039372 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5140f5410 0x7fd514105630 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fd4fc013510 tx=0x7fd4fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:23.995 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.993+0000 7fd50b7fe700 1 -- 172.21.15.174:0/137039372 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4fc009320 con 0x7fd5140f5410 2023-12-25T21:12:23.995 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.993+0000 7fd519155700 1 -- 172.21.15.174:0/137039372 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd514105d20 con 0x7fd5140f5410 2023-12-25T21:12:23.995 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.993+0000 7fd50b7fe700 1 -- 172.21.15.174:0/137039372 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4fc013f00 con 0x7fd5140f5410 2023-12-25T21:12:23.995 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.993+0000 7fd50b7fe700 1 -- 172.21.15.174:0/137039372 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4fc014d90 con 0x7fd5140f5410 2023-12-25T21:12:23.995 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.993+0000 7fd519155700 1 -- 172.21.15.174:0/137039372 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5141061b0 con 0x7fd5140f5410 2023-12-25T21:12:23.996 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.994+0000 7fd50b7fe700 1 -- 172.21.15.174:0/137039372 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd4fc0263a0 con 0x7fd5140f5410 2023-12-25T21:12:23.996 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.994+0000 7fd50b7fe700 1 --2- 172.21.15.174:0/137039372 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd50002ff60 0x7fd500032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:23.996 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.994+0000 7fd519155700 1 -- 172.21.15.174:0/137039372 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd514045ad0 con 0x7fd5140f5410 2023-12-25T21:12:23.996 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.994+0000 7fd50b7fe700 1 -- 172.21.15.174:0/137039372 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd4fc0410b0 con 0x7fd5140f5410 2023-12-25T21:12:23.996 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.994+0000 7fd51259c700 1 --2- 172.21.15.174:0/137039372 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd50002ff60 0x7fd500032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:23.996 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.994+0000 7fd51259c700 1 --2- 172.21.15.174:0/137039372 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd50002ff60 0x7fd500032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fd5040096a0 tx=0x7fd504006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:23.999 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:23.999+0000 7fd50b7fe700 1 -- 172.21.15.174:0/137039372 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd4fc012920 con 0x7fd5140f5410 2023-12-25T21:12:24.204 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:24.203+0000 7fd519155700 1 -- 172.21.15.174:0/137039372 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5140faf50 con 0x7fd5140f5410 2023-12-25T21:12:24.204 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:24.204+0000 7fd50b7fe700 1 -- 172.21.15.174:0/137039372 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd4fc02b020 con 0x7fd5140f5410 2023-12-25T21:12:24.205 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:24.205 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:24.206 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:24.205+0000 7fd519155700 1 -- 172.21.15.174:0/137039372 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd50002ff60 msgr2=0x7fd500032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:24.206 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:24.205+0000 7fd519155700 1 --2- 172.21.15.174:0/137039372 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd50002ff60 0x7fd500032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fd5040096a0 tx=0x7fd504006b40).stop 2023-12-25T21:12:24.206 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:24.205+0000 7fd519155700 1 -- 172.21.15.174:0/137039372 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5140f5410 msgr2=0x7fd514105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:24.206 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:24.205+0000 7fd519155700 1 --2- 172.21.15.174:0/137039372 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5140f5410 0x7fd514105630 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fd4fc013510 tx=0x7fd4fc00b040).stop 2023-12-25T21:12:24.206 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:24.205+0000 7fd519155700 1 -- 172.21.15.174:0/137039372 shutdown_connections 2023-12-25T21:12:24.206 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:24.205+0000 7fd519155700 1 --2- 172.21.15.174:0/137039372 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd50002ff60 0x7fd500032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:24.206 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:24.205+0000 7fd519155700 1 --2- 172.21.15.174:0/137039372 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5140f5410 0x7fd514105630 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:24.207 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:24.205+0000 7fd519155700 1 -- 172.21.15.174:0/137039372 >> 172.21.15.174:0/137039372 conn(0x7fd5140f0b70 msgr2=0x7fd5140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:24.207 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:24.205+0000 7fd519155700 1 -- 172.21.15.174:0/137039372 shutdown_connections 2023-12-25T21:12:24.207 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:24.205+0000 7fd519155700 1 -- 172.21.15.174:0/137039372 wait complete. 2023-12-25T21:12:24.207 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:24.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:24 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:22.618424+0000 mgr.smithi061.jxxeke (mgr.14164) 171 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:24.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:24 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:24.205263+0000 mon.smithi061 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.174:0/137039372' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:25.615 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:25.616 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:26.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:26 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:24.618847+0000 mgr.smithi061.jxxeke (mgr.14164) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:26.747 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.746+0000 7f52affff700 1 -- 172.21.15.174:0/356812402 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52b00f62a0 msgr2=0x7f52b00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:26.747 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.746+0000 7f52affff700 1 --2- 172.21.15.174:0/356812402 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52b00f62a0 0x7f52b00f66c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f52a0009fb0 tx=0x7f52a0009b30).stop 2023-12-25T21:12:26.747 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.746+0000 7f52affff700 1 -- 172.21.15.174:0/356812402 shutdown_connections 2023-12-25T21:12:26.747 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.746+0000 7f52affff700 1 --2- 172.21.15.174:0/356812402 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52b00f62a0 0x7f52b00f66c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:26.747 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.746+0000 7f52affff700 1 -- 172.21.15.174:0/356812402 >> 172.21.15.174:0/356812402 conn(0x7f52b00f1aa0 msgr2=0x7f52b00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:26.747 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.746+0000 7f52affff700 1 -- 172.21.15.174:0/356812402 shutdown_connections 2023-12-25T21:12:26.747 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.746+0000 7f52affff700 1 -- 172.21.15.174:0/356812402 wait complete. 2023-12-25T21:12:26.748 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.746+0000 7f52affff700 1 Processor -- start 2023-12-25T21:12:26.748 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.746+0000 7f52affff700 1 -- start start 2023-12-25T21:12:26.748 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.746+0000 7f52affff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52b00f62a0 0x7f52b01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:26.748 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.746+0000 7f52affff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b01069a0 con 0x7f52b00f62a0 2023-12-25T21:12:26.749 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.747+0000 7f52aeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52b00f62a0 0x7f52b01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:26.749 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.747+0000 7f52aeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52b00f62a0 0x7f52b01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:50320/0 (socket says 172.21.15.174:50320) 2023-12-25T21:12:26.749 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.747+0000 7f52aeffd700 1 -- 172.21.15.174:0/425990121 learned_addr learned my addr 172.21.15.174:0/425990121 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:26.749 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.747+0000 7f52aeffd700 1 -- 172.21.15.174:0/425990121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a0005040 con 0x7f52b00f62a0 2023-12-25T21:12:26.749 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.747+0000 7f52aeffd700 1 --2- 172.21.15.174:0/425990121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52b00f62a0 0x7f52b01064b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f52a0013510 tx=0x7f52a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:26.749 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.748+0000 7f52977fe700 1 -- 172.21.15.174:0/425990121 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52a0009320 con 0x7f52b00f62a0 2023-12-25T21:12:26.749 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.748+0000 7f52affff700 1 -- 172.21.15.174:0/425990121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b0106ba0 con 0x7f52b00f62a0 2023-12-25T21:12:26.750 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.748+0000 7f52affff700 1 -- 172.21.15.174:0/425990121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b0107030 con 0x7f52b00f62a0 2023-12-25T21:12:26.750 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.748+0000 7f52977fe700 1 -- 172.21.15.174:0/425990121 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52a0013f00 con 0x7f52b00f62a0 2023-12-25T21:12:26.750 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.748+0000 7f52977fe700 1 -- 172.21.15.174:0/425990121 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52a0014d90 con 0x7f52b00f62a0 2023-12-25T21:12:26.750 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.748+0000 7f52977fe700 1 -- 172.21.15.174:0/425990121 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f52a0014420 con 0x7f52b00f62a0 2023-12-25T21:12:26.750 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.749+0000 7f52977fe700 1 --2- 172.21.15.174:0/425990121 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f529802ff60 0x7f5298032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:26.750 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.749+0000 7f52affff700 1 -- 172.21.15.174:0/425990121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b0045ad0 con 0x7f52b00f62a0 2023-12-25T21:12:26.750 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.749+0000 7f52977fe700 1 -- 172.21.15.174:0/425990121 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f52a00413a0 con 0x7f52b00f62a0 2023-12-25T21:12:26.750 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.749+0000 7f52ae7fc700 1 --2- 172.21.15.174:0/425990121 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f529802ff60 0x7f5298032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:26.750 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.749+0000 7f52ae7fc700 1 --2- 172.21.15.174:0/425990121 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f529802ff60 0x7f5298032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f52a40096a0 tx=0x7f52a4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:26.754 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.754+0000 7f52977fe700 1 -- 172.21.15.174:0/425990121 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f52a001c320 con 0x7f52b00f62a0 2023-12-25T21:12:26.964 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.963+0000 7f52affff700 1 -- 172.21.15.174:0/425990121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f52b0107ee0 con 0x7f52b00f62a0 2023-12-25T21:12:26.964 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.965+0000 7f52977fe700 1 -- 172.21.15.174:0/425990121 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f52a0012b20 con 0x7f52b00f62a0 2023-12-25T21:12:26.965 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:26.965 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:26.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.966+0000 7f52affff700 1 -- 172.21.15.174:0/425990121 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f529802ff60 msgr2=0x7f5298032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:26.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.966+0000 7f52affff700 1 --2- 172.21.15.174:0/425990121 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f529802ff60 0x7f5298032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f52a40096a0 tx=0x7f52a4006b40).stop 2023-12-25T21:12:26.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.966+0000 7f52affff700 1 -- 172.21.15.174:0/425990121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52b00f62a0 msgr2=0x7f52b01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:26.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.966+0000 7f52affff700 1 --2- 172.21.15.174:0/425990121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52b00f62a0 0x7f52b01064b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f52a0013510 tx=0x7f52a000b040).stop 2023-12-25T21:12:26.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.966+0000 7f52affff700 1 -- 172.21.15.174:0/425990121 shutdown_connections 2023-12-25T21:12:26.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.966+0000 7f52affff700 1 --2- 172.21.15.174:0/425990121 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f529802ff60 0x7f5298032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:26.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.966+0000 7f52affff700 1 --2- 172.21.15.174:0/425990121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52b00f62a0 0x7f52b01064b0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:26.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.966+0000 7f52affff700 1 -- 172.21.15.174:0/425990121 >> 172.21.15.174:0/425990121 conn(0x7f52b00f1aa0 msgr2=0x7f52b00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:26.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.966+0000 7f52affff700 1 -- 172.21.15.174:0/425990121 shutdown_connections 2023-12-25T21:12:26.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:26.966+0000 7f52affff700 1 -- 172.21.15.174:0/425990121 wait complete. 2023-12-25T21:12:26.967 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:27.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:27 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:26.965320+0000 mon.smithi061 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.174:0/425990121' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:28.347 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:28.347 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:28.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:28 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:26.619065+0000 mgr.smithi061.jxxeke (mgr.14164) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:29.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.478+0000 7fd5aef92700 1 -- 172.21.15.174:0/1299487467 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a80f5410 msgr2=0x7fd5a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:29.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.478+0000 7fd5aef92700 1 --2- 172.21.15.174:0/1299487467 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a80f5410 0x7fd5a80f5830 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fd59c009fb0 tx=0x7fd59c009b30).stop 2023-12-25T21:12:29.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.478+0000 7fd5aef92700 1 -- 172.21.15.174:0/1299487467 shutdown_connections 2023-12-25T21:12:29.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.478+0000 7fd5aef92700 1 --2- 172.21.15.174:0/1299487467 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a80f5410 0x7fd5a80f5830 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:29.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.478+0000 7fd5aef92700 1 -- 172.21.15.174:0/1299487467 >> 172.21.15.174:0/1299487467 conn(0x7fd5a80f0b70 msgr2=0x7fd5a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:29.480 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.478+0000 7fd5aef92700 1 -- 172.21.15.174:0/1299487467 shutdown_connections 2023-12-25T21:12:29.480 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.478+0000 7fd5aef92700 1 -- 172.21.15.174:0/1299487467 wait complete. 2023-12-25T21:12:29.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.479+0000 7fd5aef92700 1 Processor -- start 2023-12-25T21:12:29.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.479+0000 7fd5aef92700 1 -- start start 2023-12-25T21:12:29.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.479+0000 7fd5aef92700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a80f5410 0x7fd5a8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:29.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.479+0000 7fd5aef92700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a8105b20 con 0x7fd5a80f5410 2023-12-25T21:12:29.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.480+0000 7fd5acd2e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a80f5410 0x7fd5a8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:29.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.480+0000 7fd5acd2e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a80f5410 0x7fd5a8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:50334/0 (socket says 172.21.15.174:50334) 2023-12-25T21:12:29.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.480+0000 7fd5acd2e700 1 -- 172.21.15.174:0/389749458 learned_addr learned my addr 172.21.15.174:0/389749458 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:29.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.480+0000 7fd5acd2e700 1 -- 172.21.15.174:0/389749458 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59c005040 con 0x7fd5a80f5410 2023-12-25T21:12:29.482 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.480+0000 7fd5acd2e700 1 --2- 172.21.15.174:0/389749458 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a80f5410 0x7fd5a8105630 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fd59c013510 tx=0x7fd59c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:29.482 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.480+0000 7fd5a57fa700 1 -- 172.21.15.174:0/389749458 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd59c009320 con 0x7fd5a80f5410 2023-12-25T21:12:29.482 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.480+0000 7fd5a57fa700 1 -- 172.21.15.174:0/389749458 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd59c013f00 con 0x7fd5a80f5410 2023-12-25T21:12:29.482 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.480+0000 7fd5aef92700 1 -- 172.21.15.174:0/389749458 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a8105d20 con 0x7fd5a80f5410 2023-12-25T21:12:29.482 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.480+0000 7fd5a57fa700 1 -- 172.21.15.174:0/389749458 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd59c014d90 con 0x7fd5a80f5410 2023-12-25T21:12:29.482 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.480+0000 7fd5aef92700 1 -- 172.21.15.174:0/389749458 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a81061b0 con 0x7fd5a80f5410 2023-12-25T21:12:29.483 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.481+0000 7fd5a57fa700 1 -- 172.21.15.174:0/389749458 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd59c014420 con 0x7fd5a80f5410 2023-12-25T21:12:29.483 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.481+0000 7fd5aef92700 1 -- 172.21.15.174:0/389749458 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a8045ad0 con 0x7fd5a80f5410 2023-12-25T21:12:29.483 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.481+0000 7fd5a57fa700 1 --2- 172.21.15.174:0/389749458 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd59802ff60 0x7fd598032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:29.483 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.481+0000 7fd5a57fa700 1 -- 172.21.15.174:0/389749458 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd59c0410c0 con 0x7fd5a80f5410 2023-12-25T21:12:29.483 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.481+0000 7fd5a7fff700 1 --2- 172.21.15.174:0/389749458 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd59802ff60 0x7fd598032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:29.483 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.482+0000 7fd5a7fff700 1 --2- 172.21.15.174:0/389749458 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd59802ff60 0x7fd598032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fd5940096a0 tx=0x7fd594006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:29.486 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.486+0000 7fd5a57fa700 1 -- 172.21.15.174:0/389749458 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd59c0095f0 con 0x7fd5a80f5410 2023-12-25T21:12:29.696 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.695+0000 7fd5aef92700 1 -- 172.21.15.174:0/389749458 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5a80fae60 con 0x7fd5a80f5410 2023-12-25T21:12:29.696 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.696+0000 7fd5a57fa700 1 -- 172.21.15.174:0/389749458 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd59c01db70 con 0x7fd5a80f5410 2023-12-25T21:12:29.696 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:29.697 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:29.698 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.697+0000 7fd5aef92700 1 -- 172.21.15.174:0/389749458 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd59802ff60 msgr2=0x7fd598032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:29.698 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.697+0000 7fd5aef92700 1 --2- 172.21.15.174:0/389749458 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd59802ff60 0x7fd598032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fd5940096a0 tx=0x7fd594006b40).stop 2023-12-25T21:12:29.698 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.697+0000 7fd5aef92700 1 -- 172.21.15.174:0/389749458 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a80f5410 msgr2=0x7fd5a8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:29.698 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.697+0000 7fd5aef92700 1 --2- 172.21.15.174:0/389749458 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a80f5410 0x7fd5a8105630 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fd59c013510 tx=0x7fd59c00b040).stop 2023-12-25T21:12:29.698 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.697+0000 7fd5aef92700 1 -- 172.21.15.174:0/389749458 shutdown_connections 2023-12-25T21:12:29.698 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.697+0000 7fd5aef92700 1 --2- 172.21.15.174:0/389749458 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd59802ff60 0x7fd598032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:29.698 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.697+0000 7fd5aef92700 1 --2- 172.21.15.174:0/389749458 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a80f5410 0x7fd5a8105630 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:29.698 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.697+0000 7fd5aef92700 1 -- 172.21.15.174:0/389749458 >> 172.21.15.174:0/389749458 conn(0x7fd5a80f0b70 msgr2=0x7fd5a80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:29.698 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.697+0000 7fd5aef92700 1 -- 172.21.15.174:0/389749458 shutdown_connections 2023-12-25T21:12:29.699 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:29.697+0000 7fd5aef92700 1 -- 172.21.15.174:0/389749458 wait complete. 2023-12-25T21:12:29.699 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:30.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:30 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:28.619381+0000 mgr.smithi061.jxxeke (mgr.14164) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:30.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:30 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:29.697128+0000 mon.smithi061 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.174:0/389749458' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:31.078 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:31.078 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:32.243 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.241+0000 7f5403f71700 1 -- 172.21.15.174:0/4279744106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53fc0f5410 msgr2=0x7f53fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:32.243 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.241+0000 7f5403f71700 1 --2- 172.21.15.174:0/4279744106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53fc0f5410 0x7f53fc0f5830 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f53f0009fb0 tx=0x7f53f0009b30).stop 2023-12-25T21:12:32.243 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.241+0000 7f5403f71700 1 -- 172.21.15.174:0/4279744106 shutdown_connections 2023-12-25T21:12:32.244 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.241+0000 7f5403f71700 1 --2- 172.21.15.174:0/4279744106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53fc0f5410 0x7f53fc0f5830 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:32.244 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.241+0000 7f5403f71700 1 -- 172.21.15.174:0/4279744106 >> 172.21.15.174:0/4279744106 conn(0x7f53fc0f0b70 msgr2=0x7f53fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:32.244 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.242+0000 7f5403f71700 1 -- 172.21.15.174:0/4279744106 shutdown_connections 2023-12-25T21:12:32.244 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.242+0000 7f5403f71700 1 -- 172.21.15.174:0/4279744106 wait complete. 2023-12-25T21:12:32.244 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.242+0000 7f5403f71700 1 Processor -- start 2023-12-25T21:12:32.244 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.242+0000 7f5403f71700 1 -- start start 2023-12-25T21:12:32.244 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.242+0000 7f5403f71700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53fc0f5410 0x7f53fc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:32.244 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.242+0000 7f5403f71700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc106c20 con 0x7f53fc0f5410 2023-12-25T21:12:32.244 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.242+0000 7f5401d0d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53fc0f5410 0x7f53fc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:32.244 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.243+0000 7f5401d0d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53fc0f5410 0x7f53fc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:50350/0 (socket says 172.21.15.174:50350) 2023-12-25T21:12:32.244 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.243+0000 7f5401d0d700 1 -- 172.21.15.174:0/2226380660 learned_addr learned my addr 172.21.15.174:0/2226380660 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:32.244 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.243+0000 7f5401d0d700 1 -- 172.21.15.174:0/2226380660 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f0005040 con 0x7f53fc0f5410 2023-12-25T21:12:32.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.243+0000 7f5401d0d700 1 --2- 172.21.15.174:0/2226380660 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53fc0f5410 0x7f53fc106730 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f53f0014000 tx=0x7f53f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:32.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.243+0000 7f53ee7fc700 1 -- 172.21.15.174:0/2226380660 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53f00095f0 con 0x7f53fc0f5410 2023-12-25T21:12:32.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.243+0000 7f5403f71700 1 -- 172.21.15.174:0/2226380660 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53fc106e20 con 0x7f53fc0f5410 2023-12-25T21:12:32.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.243+0000 7f53ee7fc700 1 -- 172.21.15.174:0/2226380660 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53f0007e70 con 0x7f53fc0f5410 2023-12-25T21:12:32.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.243+0000 7f5403f71700 1 -- 172.21.15.174:0/2226380660 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53fc1072b0 con 0x7f53fc0f5410 2023-12-25T21:12:32.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.243+0000 7f53ee7fc700 1 -- 172.21.15.174:0/2226380660 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53f0014e90 con 0x7f53fc0f5410 2023-12-25T21:12:32.246 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.244+0000 7f53ee7fc700 1 -- 172.21.15.174:0/2226380660 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f53f0014690 con 0x7f53fc0f5410 2023-12-25T21:12:32.246 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.244+0000 7f5403f71700 1 -- 172.21.15.174:0/2226380660 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53fc045ad0 con 0x7f53fc0f5410 2023-12-25T21:12:32.246 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.244+0000 7f53ee7fc700 1 --2- 172.21.15.174:0/2226380660 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f53e802ff60 0x7f53e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:32.246 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.244+0000 7f53ee7fc700 1 -- 172.21.15.174:0/2226380660 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f53f0040fb0 con 0x7f53fc0f5410 2023-12-25T21:12:32.246 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.244+0000 7f540150c700 1 --2- 172.21.15.174:0/2226380660 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f53e802ff60 0x7f53e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:32.246 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.245+0000 7f540150c700 1 --2- 172.21.15.174:0/2226380660 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f53e802ff60 0x7f53e8032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f53f80096a0 tx=0x7f53f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:32.248 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.249+0000 7f53ee7fc700 1 -- 172.21.15.174:0/2226380660 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53f006d2b0 con 0x7f53fc0f5410 2023-12-25T21:12:32.456 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.455+0000 7f5403f71700 1 -- 172.21.15.174:0/2226380660 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53fc0fae60 con 0x7f53fc0f5410 2023-12-25T21:12:32.456 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.456+0000 7f53ee7fc700 1 -- 172.21.15.174:0/2226380660 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f53f0012980 con 0x7f53fc0f5410 2023-12-25T21:12:32.456 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:32.456 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:32.457 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.457+0000 7f5403f71700 1 -- 172.21.15.174:0/2226380660 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f53e802ff60 msgr2=0x7f53e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:32.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.457+0000 7f5403f71700 1 --2- 172.21.15.174:0/2226380660 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f53e802ff60 0x7f53e8032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f53f80096a0 tx=0x7f53f8006b40).stop 2023-12-25T21:12:32.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.457+0000 7f5403f71700 1 -- 172.21.15.174:0/2226380660 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53fc0f5410 msgr2=0x7f53fc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:32.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.457+0000 7f5403f71700 1 --2- 172.21.15.174:0/2226380660 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53fc0f5410 0x7f53fc106730 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f53f0014000 tx=0x7f53f000b040).stop 2023-12-25T21:12:32.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.458+0000 7f5403f71700 1 -- 172.21.15.174:0/2226380660 shutdown_connections 2023-12-25T21:12:32.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.458+0000 7f5403f71700 1 --2- 172.21.15.174:0/2226380660 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f53e802ff60 0x7f53e8032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:32.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.458+0000 7f5403f71700 1 --2- 172.21.15.174:0/2226380660 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53fc0f5410 0x7f53fc106730 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:32.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.458+0000 7f5403f71700 1 -- 172.21.15.174:0/2226380660 >> 172.21.15.174:0/2226380660 conn(0x7f53fc0f0b70 msgr2=0x7f53fc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:32.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.458+0000 7f5403f71700 1 -- 172.21.15.174:0/2226380660 shutdown_connections 2023-12-25T21:12:32.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:32.458+0000 7f5403f71700 1 -- 172.21.15.174:0/2226380660 wait complete. 2023-12-25T21:12:32.458 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:32.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:32 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:30.619709+0000 mgr.smithi061.jxxeke (mgr.14164) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:33.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:33 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:32.456474+0000 mon.smithi061 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.174:0/2226380660' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:33.868 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:33.868 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:34.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:34 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:32.620046+0000 mgr.smithi061.jxxeke (mgr.14164) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:35.015 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.015+0000 7fd0834e1700 1 -- 172.21.15.174:0/294259247 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd07c0f5410 msgr2=0x7fd07c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:35.015 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.015+0000 7fd0834e1700 1 --2- 172.21.15.174:0/294259247 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd07c0f5410 0x7fd07c0f5830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fd078009fb0 tx=0x7fd078009b30).stop 2023-12-25T21:12:35.015 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.015+0000 7fd0834e1700 1 -- 172.21.15.174:0/294259247 shutdown_connections 2023-12-25T21:12:35.015 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.015+0000 7fd0834e1700 1 --2- 172.21.15.174:0/294259247 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd07c0f5410 0x7fd07c0f5830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:35.015 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.015+0000 7fd0834e1700 1 -- 172.21.15.174:0/294259247 >> 172.21.15.174:0/294259247 conn(0x7fd07c0f0b70 msgr2=0x7fd07c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:35.015 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.015+0000 7fd0834e1700 1 -- 172.21.15.174:0/294259247 shutdown_connections 2023-12-25T21:12:35.015 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.015+0000 7fd0834e1700 1 -- 172.21.15.174:0/294259247 wait complete. 2023-12-25T21:12:35.017 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.016+0000 7fd0834e1700 1 Processor -- start 2023-12-25T21:12:35.017 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.016+0000 7fd0834e1700 1 -- start start 2023-12-25T21:12:35.017 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.016+0000 7fd0834e1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd07c0f5410 0x7fd07c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:35.017 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.016+0000 7fd0834e1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c106c20 con 0x7fd07c0f5410 2023-12-25T21:12:35.017 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.016+0000 7fd08127d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd07c0f5410 0x7fd07c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:35.017 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.016+0000 7fd08127d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd07c0f5410 0x7fd07c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:50362/0 (socket says 172.21.15.174:50362) 2023-12-25T21:12:35.017 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.016+0000 7fd08127d700 1 -- 172.21.15.174:0/1823354514 learned_addr learned my addr 172.21.15.174:0/1823354514 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:35.017 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.016+0000 7fd08127d700 1 -- 172.21.15.174:0/1823354514 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd078005040 con 0x7fd07c0f5410 2023-12-25T21:12:35.017 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.016+0000 7fd08127d700 1 --2- 172.21.15.174:0/1823354514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd07c0f5410 0x7fd07c106730 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fd078013510 tx=0x7fd07800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:35.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.017+0000 7fd071ffb700 1 -- 172.21.15.174:0/1823354514 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd078009320 con 0x7fd07c0f5410 2023-12-25T21:12:35.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.017+0000 7fd071ffb700 1 -- 172.21.15.174:0/1823354514 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd078013f00 con 0x7fd07c0f5410 2023-12-25T21:12:35.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.017+0000 7fd071ffb700 1 -- 172.21.15.174:0/1823354514 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd078014d90 con 0x7fd07c0f5410 2023-12-25T21:12:35.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.017+0000 7fd0834e1700 1 -- 172.21.15.174:0/1823354514 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd07c106e20 con 0x7fd07c0f5410 2023-12-25T21:12:35.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.017+0000 7fd0834e1700 1 -- 172.21.15.174:0/1823354514 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd07c1072b0 con 0x7fd07c0f5410 2023-12-25T21:12:35.019 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.018+0000 7fd071ffb700 1 -- 172.21.15.174:0/1823354514 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd0780095f0 con 0x7fd07c0f5410 2023-12-25T21:12:35.019 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.018+0000 7fd0834e1700 1 -- 172.21.15.174:0/1823354514 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd07c045ad0 con 0x7fd07c0f5410 2023-12-25T21:12:35.019 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.018+0000 7fd071ffb700 1 --2- 172.21.15.174:0/1823354514 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd06802ffb0 0x7fd068032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:35.019 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.018+0000 7fd071ffb700 1 -- 172.21.15.174:0/1823354514 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd0780410c0 con 0x7fd07c0f5410 2023-12-25T21:12:35.019 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.018+0000 7fd080a7c700 1 --2- 172.21.15.174:0/1823354514 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd06802ffb0 0x7fd068032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:35.019 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.019+0000 7fd080a7c700 1 --2- 172.21.15.174:0/1823354514 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd06802ffb0 0x7fd068032470 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fd06c0096a0 tx=0x7fd06c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:35.022 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.023+0000 7fd071ffb700 1 -- 172.21.15.174:0/1823354514 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd07801d7f0 con 0x7fd07c0f5410 2023-12-25T21:12:35.232 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.232+0000 7fd0834e1700 1 -- 172.21.15.174:0/1823354514 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd07c0fae60 con 0x7fd07c0f5410 2023-12-25T21:12:35.232 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.232+0000 7fd071ffb700 1 -- 172.21.15.174:0/1823354514 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd078010230 con 0x7fd07c0f5410 2023-12-25T21:12:35.232 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:35.232 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:35.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.234+0000 7fd0834e1700 1 -- 172.21.15.174:0/1823354514 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd06802ffb0 msgr2=0x7fd068032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:35.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.234+0000 7fd0834e1700 1 --2- 172.21.15.174:0/1823354514 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd06802ffb0 0x7fd068032470 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fd06c0096a0 tx=0x7fd06c006b40).stop 2023-12-25T21:12:35.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.234+0000 7fd0834e1700 1 -- 172.21.15.174:0/1823354514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd07c0f5410 msgr2=0x7fd07c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:35.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.234+0000 7fd0834e1700 1 --2- 172.21.15.174:0/1823354514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd07c0f5410 0x7fd07c106730 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fd078013510 tx=0x7fd07800b040).stop 2023-12-25T21:12:35.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.234+0000 7fd0834e1700 1 -- 172.21.15.174:0/1823354514 shutdown_connections 2023-12-25T21:12:35.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.234+0000 7fd0834e1700 1 --2- 172.21.15.174:0/1823354514 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd06802ffb0 0x7fd068032470 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:35.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.234+0000 7fd0834e1700 1 --2- 172.21.15.174:0/1823354514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd07c0f5410 0x7fd07c106730 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:35.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.234+0000 7fd0834e1700 1 -- 172.21.15.174:0/1823354514 >> 172.21.15.174:0/1823354514 conn(0x7fd07c0f0b70 msgr2=0x7fd07c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:35.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.234+0000 7fd0834e1700 1 -- 172.21.15.174:0/1823354514 shutdown_connections 2023-12-25T21:12:35.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:35.234+0000 7fd0834e1700 1 -- 172.21.15.174:0/1823354514 wait complete. 2023-12-25T21:12:35.235 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:35.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:35 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:35.232721+0000 mon.smithi061 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.174:0/1823354514' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:36.620 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:36.620 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:36.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:36 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:34.620282+0000 mgr.smithi061.jxxeke (mgr.14164) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:37.740 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.739+0000 7ffab4ad7700 1 -- 172.21.15.174:0/638961389 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffab00f6510 msgr2=0x7ffab00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:37.741 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.739+0000 7ffab4ad7700 1 --2- 172.21.15.174:0/638961389 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffab00f6510 0x7ffab00f6930 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7ffaa4009fb0 tx=0x7ffaa4009b30).stop 2023-12-25T21:12:37.741 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.739+0000 7ffab4ad7700 1 -- 172.21.15.174:0/638961389 shutdown_connections 2023-12-25T21:12:37.741 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.739+0000 7ffab4ad7700 1 --2- 172.21.15.174:0/638961389 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffab00f6510 0x7ffab00f6930 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:37.741 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.739+0000 7ffab4ad7700 1 -- 172.21.15.174:0/638961389 >> 172.21.15.174:0/638961389 conn(0x7ffab00f1c70 msgr2=0x7ffab00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:37.741 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.739+0000 7ffab4ad7700 1 -- 172.21.15.174:0/638961389 shutdown_connections 2023-12-25T21:12:37.741 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.739+0000 7ffab4ad7700 1 -- 172.21.15.174:0/638961389 wait complete. 2023-12-25T21:12:37.741 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.739+0000 7ffab4ad7700 1 Processor -- start 2023-12-25T21:12:37.741 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.740+0000 7ffab4ad7700 1 -- start start 2023-12-25T21:12:37.741 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.740+0000 7ffab4ad7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffab00f6510 0x7ffab0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:37.741 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.740+0000 7ffab4ad7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab0106c20 con 0x7ffab00f6510 2023-12-25T21:12:37.741 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.740+0000 7ffaae59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffab00f6510 0x7ffab0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:37.741 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.740+0000 7ffaae59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffab00f6510 0x7ffab0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:43564/0 (socket says 172.21.15.174:43564) 2023-12-25T21:12:37.742 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.740+0000 7ffaae59c700 1 -- 172.21.15.174:0/3065542255 learned_addr learned my addr 172.21.15.174:0/3065542255 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:37.742 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.740+0000 7ffaae59c700 1 -- 172.21.15.174:0/3065542255 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa4005040 con 0x7ffab00f6510 2023-12-25T21:12:37.742 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.740+0000 7ffaae59c700 1 --2- 172.21.15.174:0/3065542255 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffab00f6510 0x7ffab0106730 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7ffaa4013510 tx=0x7ffaa400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:37.742 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.741+0000 7ffa9effd700 1 -- 172.21.15.174:0/3065542255 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffaa4009320 con 0x7ffab00f6510 2023-12-25T21:12:37.742 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.741+0000 7ffab4ad7700 1 -- 172.21.15.174:0/3065542255 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab0106e20 con 0x7ffab00f6510 2023-12-25T21:12:37.742 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.741+0000 7ffa9effd700 1 -- 172.21.15.174:0/3065542255 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffaa4013f00 con 0x7ffab00f6510 2023-12-25T21:12:37.742 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.741+0000 7ffa9effd700 1 -- 172.21.15.174:0/3065542255 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffaa4014d90 con 0x7ffab00f6510 2023-12-25T21:12:37.742 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.741+0000 7ffab4ad7700 1 -- 172.21.15.174:0/3065542255 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab01072b0 con 0x7ffab00f6510 2023-12-25T21:12:37.743 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.741+0000 7ffa9effd700 1 -- 172.21.15.174:0/3065542255 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffaa40095f0 con 0x7ffab00f6510 2023-12-25T21:12:37.743 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.741+0000 7ffab4ad7700 1 -- 172.21.15.174:0/3065542255 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab0045ad0 con 0x7ffab00f6510 2023-12-25T21:12:37.743 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.742+0000 7ffa9effd700 1 --2- 172.21.15.174:0/3065542255 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ffa9802ffb0 0x7ffa98032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:37.743 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.742+0000 7ffa9effd700 1 -- 172.21.15.174:0/3065542255 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffaa40410a0 con 0x7ffab00f6510 2023-12-25T21:12:37.743 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.742+0000 7ffaadd9b700 1 --2- 172.21.15.174:0/3065542255 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ffa9802ffb0 0x7ffa98032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:37.743 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.742+0000 7ffaadd9b700 1 --2- 172.21.15.174:0/3065542255 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ffa9802ffb0 0x7ffa98032470 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7ffaa00096a0 tx=0x7ffaa0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:37.746 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.747+0000 7ffa9effd700 1 -- 172.21.15.174:0/3065542255 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffaa406d2b0 con 0x7ffab00f6510 2023-12-25T21:12:37.952 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.952+0000 7ffab4ad7700 1 -- 172.21.15.174:0/3065542255 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffab00fbf60 con 0x7ffab00f6510 2023-12-25T21:12:37.953 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.953+0000 7ffa9effd700 1 -- 172.21.15.174:0/3065542255 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffaa401d7f0 con 0x7ffab00f6510 2023-12-25T21:12:37.953 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:37.953 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:37.954 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.955+0000 7ffab4ad7700 1 -- 172.21.15.174:0/3065542255 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ffa9802ffb0 msgr2=0x7ffa98032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:37.955 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.955+0000 7ffab4ad7700 1 --2- 172.21.15.174:0/3065542255 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ffa9802ffb0 0x7ffa98032470 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7ffaa00096a0 tx=0x7ffaa0006b40).stop 2023-12-25T21:12:37.955 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.955+0000 7ffab4ad7700 1 -- 172.21.15.174:0/3065542255 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffab00f6510 msgr2=0x7ffab0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:37.955 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.955+0000 7ffab4ad7700 1 --2- 172.21.15.174:0/3065542255 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffab00f6510 0x7ffab0106730 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7ffaa4013510 tx=0x7ffaa400b040).stop 2023-12-25T21:12:37.955 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.955+0000 7ffab4ad7700 1 -- 172.21.15.174:0/3065542255 shutdown_connections 2023-12-25T21:12:37.955 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.955+0000 7ffab4ad7700 1 --2- 172.21.15.174:0/3065542255 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ffa9802ffb0 0x7ffa98032470 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:37.955 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.955+0000 7ffab4ad7700 1 --2- 172.21.15.174:0/3065542255 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffab00f6510 0x7ffab0106730 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:37.955 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.955+0000 7ffab4ad7700 1 -- 172.21.15.174:0/3065542255 >> 172.21.15.174:0/3065542255 conn(0x7ffab00f1c70 msgr2=0x7ffab00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:37.955 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.955+0000 7ffab4ad7700 1 -- 172.21.15.174:0/3065542255 shutdown_connections 2023-12-25T21:12:37.955 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:37.955+0000 7ffab4ad7700 1 -- 172.21.15.174:0/3065542255 wait complete. 2023-12-25T21:12:37.955 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:38.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:38 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:36.620649+0000 mgr.smithi061.jxxeke (mgr.14164) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:38.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:38 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:37.953382+0000 mon.smithi061 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.174:0/3065542255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:39.343 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:39.344 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:40.566 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.565+0000 7fca3bf31700 1 -- 172.21.15.174:0/929233550 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca340f5410 msgr2=0x7fca340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:40.566 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.565+0000 7fca3bf31700 1 --2- 172.21.15.174:0/929233550 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca340f5410 0x7fca340f5830 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fca24009fb0 tx=0x7fca24009b30).stop 2023-12-25T21:12:40.566 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.565+0000 7fca3bf31700 1 -- 172.21.15.174:0/929233550 shutdown_connections 2023-12-25T21:12:40.566 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.565+0000 7fca3bf31700 1 --2- 172.21.15.174:0/929233550 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca340f5410 0x7fca340f5830 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:40.567 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.565+0000 7fca3bf31700 1 -- 172.21.15.174:0/929233550 >> 172.21.15.174:0/929233550 conn(0x7fca340f0b70 msgr2=0x7fca340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:40.567 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.565+0000 7fca3bf31700 1 -- 172.21.15.174:0/929233550 shutdown_connections 2023-12-25T21:12:40.567 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.565+0000 7fca3bf31700 1 -- 172.21.15.174:0/929233550 wait complete. 2023-12-25T21:12:40.567 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.566+0000 7fca3bf31700 1 Processor -- start 2023-12-25T21:12:40.567 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.566+0000 7fca3bf31700 1 -- start start 2023-12-25T21:12:40.567 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.566+0000 7fca3bf31700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca340f5410 0x7fca34109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:40.567 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.566+0000 7fca3bf31700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca34060240 con 0x7fca340f5410 2023-12-25T21:12:40.567 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.566+0000 7fca39ccd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca340f5410 0x7fca34109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:40.567 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.566+0000 7fca39ccd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca340f5410 0x7fca34109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:43574/0 (socket says 172.21.15.174:43574) 2023-12-25T21:12:40.567 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.566+0000 7fca39ccd700 1 -- 172.21.15.174:0/877359996 learned_addr learned my addr 172.21.15.174:0/877359996 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:40.567 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.567+0000 7fca39ccd700 1 -- 172.21.15.174:0/877359996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca24005040 con 0x7fca340f5410 2023-12-25T21:12:40.569 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.567+0000 7fca39ccd700 1 --2- 172.21.15.174:0/877359996 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca340f5410 0x7fca34109400 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fca24013510 tx=0x7fca2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:40.569 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.567+0000 7fca2a7fc700 1 -- 172.21.15.174:0/877359996 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca24009320 con 0x7fca340f5410 2023-12-25T21:12:40.569 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.567+0000 7fca3bf31700 1 -- 172.21.15.174:0/877359996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca34060440 con 0x7fca340f5410 2023-12-25T21:12:40.569 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.567+0000 7fca2a7fc700 1 -- 172.21.15.174:0/877359996 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca24013f00 con 0x7fca340f5410 2023-12-25T21:12:40.569 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.567+0000 7fca2a7fc700 1 -- 172.21.15.174:0/877359996 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca24014d90 con 0x7fca340f5410 2023-12-25T21:12:40.569 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.567+0000 7fca3bf31700 1 -- 172.21.15.174:0/877359996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca34109ab0 con 0x7fca340f5410 2023-12-25T21:12:40.569 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.568+0000 7fca2a7fc700 1 -- 172.21.15.174:0/877359996 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca24014420 con 0x7fca340f5410 2023-12-25T21:12:40.569 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.568+0000 7fca3bf31700 1 -- 172.21.15.174:0/877359996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca34045ad0 con 0x7fca340f5410 2023-12-25T21:12:40.569 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.568+0000 7fca2a7fc700 1 --2- 172.21.15.174:0/877359996 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fca2002ff60 0x7fca20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:40.569 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.568+0000 7fca2a7fc700 1 -- 172.21.15.174:0/877359996 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca24041000 con 0x7fca340f5410 2023-12-25T21:12:40.569 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.568+0000 7fca394cc700 1 --2- 172.21.15.174:0/877359996 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fca2002ff60 0x7fca20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:40.569 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.569+0000 7fca394cc700 1 --2- 172.21.15.174:0/877359996 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fca2002ff60 0x7fca20032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fca300096a0 tx=0x7fca30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:40.572 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.573+0000 7fca2a7fc700 1 -- 172.21.15.174:0/877359996 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca240095f0 con 0x7fca340f5410 2023-12-25T21:12:40.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:40 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:38.621047+0000 mgr.smithi061.jxxeke (mgr.14164) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:40.777 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.777+0000 7fca3bf31700 1 -- 172.21.15.174:0/877359996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca340fae60 con 0x7fca340f5410 2023-12-25T21:12:40.777 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.778+0000 7fca2a7fc700 1 -- 172.21.15.174:0/877359996 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fca2401db70 con 0x7fca340f5410 2023-12-25T21:12:40.777 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:40.777 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:40.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.779+0000 7fca3bf31700 1 -- 172.21.15.174:0/877359996 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fca2002ff60 msgr2=0x7fca20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:40.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.779+0000 7fca3bf31700 1 --2- 172.21.15.174:0/877359996 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fca2002ff60 0x7fca20032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fca300096a0 tx=0x7fca30006b40).stop 2023-12-25T21:12:40.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.779+0000 7fca3bf31700 1 -- 172.21.15.174:0/877359996 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca340f5410 msgr2=0x7fca34109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:40.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.779+0000 7fca3bf31700 1 --2- 172.21.15.174:0/877359996 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca340f5410 0x7fca34109400 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fca24013510 tx=0x7fca2400b040).stop 2023-12-25T21:12:40.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.779+0000 7fca3bf31700 1 -- 172.21.15.174:0/877359996 shutdown_connections 2023-12-25T21:12:40.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.779+0000 7fca3bf31700 1 --2- 172.21.15.174:0/877359996 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fca2002ff60 0x7fca20032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:40.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.779+0000 7fca3bf31700 1 --2- 172.21.15.174:0/877359996 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca340f5410 0x7fca34109400 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:40.779 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.779+0000 7fca3bf31700 1 -- 172.21.15.174:0/877359996 >> 172.21.15.174:0/877359996 conn(0x7fca340f0b70 msgr2=0x7fca340f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:40.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.779+0000 7fca3bf31700 1 -- 172.21.15.174:0/877359996 shutdown_connections 2023-12-25T21:12:40.780 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:40.779+0000 7fca3bf31700 1 -- 172.21.15.174:0/877359996 wait complete. 2023-12-25T21:12:40.780 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:41.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:41 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:40.778087+0000 mon.smithi061 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.174:0/877359996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:42.175 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:42.175 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:42.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:42 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:40.621412+0000 mgr.smithi061.jxxeke (mgr.14164) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:43.281 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.280+0000 7f221ae9f700 1 -- 172.21.15.174:0/830885995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22140f51a0 msgr2=0x7f22140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:43.281 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.280+0000 7f221ae9f700 1 --2- 172.21.15.174:0/830885995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22140f51a0 0x7f22140f55c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f2204009fb0 tx=0x7f2204009b30).stop 2023-12-25T21:12:43.281 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.280+0000 7f221ae9f700 1 -- 172.21.15.174:0/830885995 shutdown_connections 2023-12-25T21:12:43.281 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.280+0000 7f221ae9f700 1 --2- 172.21.15.174:0/830885995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22140f51a0 0x7f22140f55c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:43.281 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.280+0000 7f221ae9f700 1 -- 172.21.15.174:0/830885995 >> 172.21.15.174:0/830885995 conn(0x7f22140f09a0 msgr2=0x7f22140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:43.282 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.280+0000 7f221ae9f700 1 -- 172.21.15.174:0/830885995 shutdown_connections 2023-12-25T21:12:43.282 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.280+0000 7f221ae9f700 1 -- 172.21.15.174:0/830885995 wait complete. 2023-12-25T21:12:43.282 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.280+0000 7f221ae9f700 1 Processor -- start 2023-12-25T21:12:43.282 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.280+0000 7f221ae9f700 1 -- start start 2023-12-25T21:12:43.282 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.281+0000 7f221ae9f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22140f51a0 0x7f22141091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:43.282 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.281+0000 7f221ae9f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2214060240 con 0x7f22140f51a0 2023-12-25T21:12:43.282 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.281+0000 7f2219e9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22140f51a0 0x7f22141091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:43.282 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.281+0000 7f2219e9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22140f51a0 0x7f22141091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:43584/0 (socket says 172.21.15.174:43584) 2023-12-25T21:12:43.282 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.281+0000 7f2219e9d700 1 -- 172.21.15.174:0/1342683946 learned_addr learned my addr 172.21.15.174:0/1342683946 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:43.282 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.281+0000 7f2219e9d700 1 -- 172.21.15.174:0/1342683946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2204005040 con 0x7f22140f51a0 2023-12-25T21:12:43.282 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.281+0000 7f2219e9d700 1 --2- 172.21.15.174:0/1342683946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22140f51a0 0x7f22141091b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f2204013510 tx=0x7f220400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:43.282 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.281+0000 7f220a7fc700 1 -- 172.21.15.174:0/1342683946 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2204009320 con 0x7f22140f51a0 2023-12-25T21:12:43.282 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.281+0000 7f221ae9f700 1 -- 172.21.15.174:0/1342683946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2214060440 con 0x7f22140f51a0 2023-12-25T21:12:43.282 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.281+0000 7f220a7fc700 1 -- 172.21.15.174:0/1342683946 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2204013f00 con 0x7f22140f51a0 2023-12-25T21:12:43.282 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.281+0000 7f220a7fc700 1 -- 172.21.15.174:0/1342683946 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2204014dc0 con 0x7f22140f51a0 2023-12-25T21:12:43.283 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.281+0000 7f221ae9f700 1 -- 172.21.15.174:0/1342683946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2214109860 con 0x7f22140f51a0 2023-12-25T21:12:43.283 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.282+0000 7f220a7fc700 1 -- 172.21.15.174:0/1342683946 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2204014420 con 0x7f22140f51a0 2023-12-25T21:12:43.283 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.282+0000 7f221ae9f700 1 -- 172.21.15.174:0/1342683946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2214045ad0 con 0x7f22140f51a0 2023-12-25T21:12:43.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.282+0000 7f220a7fc700 1 --2- 172.21.15.174:0/1342683946 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f220002ff60 0x7f2200032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:43.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.282+0000 7f220a7fc700 1 -- 172.21.15.174:0/1342683946 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2204041030 con 0x7f22140f51a0 2023-12-25T21:12:43.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.283+0000 7f221969c700 1 --2- 172.21.15.174:0/1342683946 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f220002ff60 0x7f2200032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:43.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.283+0000 7f221969c700 1 --2- 172.21.15.174:0/1342683946 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f220002ff60 0x7f2200032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f22100096a0 tx=0x7f2210006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:43.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.288+0000 7f220a7fc700 1 -- 172.21.15.174:0/1342683946 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f22040095f0 con 0x7f22140f51a0 2023-12-25T21:12:43.495 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.495+0000 7f221ae9f700 1 -- 172.21.15.174:0/1342683946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f221410a080 con 0x7f22140f51a0 2023-12-25T21:12:43.496 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.496+0000 7f220a7fc700 1 -- 172.21.15.174:0/1342683946 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f220401cb10 con 0x7f22140f51a0 2023-12-25T21:12:43.496 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:43.497 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:43.498 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.497+0000 7f221ae9f700 1 -- 172.21.15.174:0/1342683946 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f220002ff60 msgr2=0x7f2200032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:43.498 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.497+0000 7f221ae9f700 1 --2- 172.21.15.174:0/1342683946 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f220002ff60 0x7f2200032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f22100096a0 tx=0x7f2210006b40).stop 2023-12-25T21:12:43.498 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.497+0000 7f221ae9f700 1 -- 172.21.15.174:0/1342683946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22140f51a0 msgr2=0x7f22141091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:43.498 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.497+0000 7f221ae9f700 1 --2- 172.21.15.174:0/1342683946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22140f51a0 0x7f22141091b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f2204013510 tx=0x7f220400b040).stop 2023-12-25T21:12:43.498 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.497+0000 7f221ae9f700 1 -- 172.21.15.174:0/1342683946 shutdown_connections 2023-12-25T21:12:43.499 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.497+0000 7f221ae9f700 1 --2- 172.21.15.174:0/1342683946 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f220002ff60 0x7f2200032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:43.499 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.497+0000 7f221ae9f700 1 --2- 172.21.15.174:0/1342683946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22140f51a0 0x7f22141091b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:43.499 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.497+0000 7f221ae9f700 1 -- 172.21.15.174:0/1342683946 >> 172.21.15.174:0/1342683946 conn(0x7f22140f09a0 msgr2=0x7f22140f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:43.499 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.497+0000 7f221ae9f700 1 -- 172.21.15.174:0/1342683946 shutdown_connections 2023-12-25T21:12:43.499 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:43.497+0000 7f221ae9f700 1 -- 172.21.15.174:0/1342683946 wait complete. 2023-12-25T21:12:43.499 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:44.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:44 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:42.621764+0000 mgr.smithi061.jxxeke (mgr.14164) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:44.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:44 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:43.496275+0000 mon.smithi061 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.174:0/1342683946' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:44.879 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:44.879 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:46.028 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.026+0000 7fc126828700 1 -- 172.21.15.174:0/1653873725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1200f5410 msgr2=0x7fc1200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:46.029 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.026+0000 7fc126828700 1 --2- 172.21.15.174:0/1653873725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1200f5410 0x7fc1200f5830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fc110009fb0 tx=0x7fc110009b30).stop 2023-12-25T21:12:46.029 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.026+0000 7fc126828700 1 -- 172.21.15.174:0/1653873725 shutdown_connections 2023-12-25T21:12:46.029 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.026+0000 7fc126828700 1 --2- 172.21.15.174:0/1653873725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1200f5410 0x7fc1200f5830 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:46.029 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.026+0000 7fc126828700 1 -- 172.21.15.174:0/1653873725 >> 172.21.15.174:0/1653873725 conn(0x7fc1200f0b70 msgr2=0x7fc1200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:46.030 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.026+0000 7fc126828700 1 -- 172.21.15.174:0/1653873725 shutdown_connections 2023-12-25T21:12:46.030 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.026+0000 7fc126828700 1 -- 172.21.15.174:0/1653873725 wait complete. 2023-12-25T21:12:46.030 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.027+0000 7fc126828700 1 Processor -- start 2023-12-25T21:12:46.030 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.027+0000 7fc126828700 1 -- start start 2023-12-25T21:12:46.030 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.027+0000 7fc126828700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1200f5410 0x7fc120109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:46.030 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.027+0000 7fc126828700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc120060240 con 0x7fc1200f5410 2023-12-25T21:12:46.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.027+0000 7fc11ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1200f5410 0x7fc120109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:46.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.027+0000 7fc11ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1200f5410 0x7fc120109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:43610/0 (socket says 172.21.15.174:43610) 2023-12-25T21:12:46.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.027+0000 7fc11ffff700 1 -- 172.21.15.174:0/1085155941 learned_addr learned my addr 172.21.15.174:0/1085155941 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:46.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.028+0000 7fc11ffff700 1 -- 172.21.15.174:0/1085155941 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc110005040 con 0x7fc1200f5410 2023-12-25T21:12:46.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.028+0000 7fc11ffff700 1 --2- 172.21.15.174:0/1085155941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1200f5410 0x7fc120109400 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fc110013510 tx=0x7fc11000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:46.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.028+0000 7fc11cff9700 1 -- 172.21.15.174:0/1085155941 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc110009320 con 0x7fc1200f5410 2023-12-25T21:12:46.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.028+0000 7fc126828700 1 -- 172.21.15.174:0/1085155941 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc120060440 con 0x7fc1200f5410 2023-12-25T21:12:46.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.028+0000 7fc11cff9700 1 -- 172.21.15.174:0/1085155941 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc110013f00 con 0x7fc1200f5410 2023-12-25T21:12:46.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.028+0000 7fc11cff9700 1 -- 172.21.15.174:0/1085155941 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc110014d90 con 0x7fc1200f5410 2023-12-25T21:12:46.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.028+0000 7fc126828700 1 -- 172.21.15.174:0/1085155941 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc120109ab0 con 0x7fc1200f5410 2023-12-25T21:12:46.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.029+0000 7fc11cff9700 1 -- 172.21.15.174:0/1085155941 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc1100263a0 con 0x7fc1200f5410 2023-12-25T21:12:46.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.029+0000 7fc126828700 1 -- 172.21.15.174:0/1085155941 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc120045ad0 con 0x7fc1200f5410 2023-12-25T21:12:46.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.029+0000 7fc11cff9700 1 --2- 172.21.15.174:0/1085155941 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fc10802ff60 0x7fc108032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:46.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.030+0000 7fc11cff9700 1 -- 172.21.15.174:0/1085155941 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc1100412c0 con 0x7fc1200f5410 2023-12-25T21:12:46.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.030+0000 7fc11f7fe700 1 --2- 172.21.15.174:0/1085155941 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fc10802ff60 0x7fc108032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:46.035 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.030+0000 7fc11f7fe700 1 --2- 172.21.15.174:0/1085155941 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fc10802ff60 0x7fc108032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc1140096a0 tx=0x7fc114006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:46.035 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.034+0000 7fc11cff9700 1 -- 172.21.15.174:0/1085155941 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc110012330 con 0x7fc1200f5410 2023-12-25T21:12:46.242 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.243+0000 7fc126828700 1 -- 172.21.15.174:0/1085155941 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1200fae60 con 0x7fc1200f5410 2023-12-25T21:12:46.243 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.243+0000 7fc11cff9700 1 -- 172.21.15.174:0/1085155941 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc110012330 con 0x7fc1200f5410 2023-12-25T21:12:46.243 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:46.243 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:46.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.245+0000 7fc126828700 1 -- 172.21.15.174:0/1085155941 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fc10802ff60 msgr2=0x7fc108032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:46.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.245+0000 7fc126828700 1 --2- 172.21.15.174:0/1085155941 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fc10802ff60 0x7fc108032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc1140096a0 tx=0x7fc114006b40).stop 2023-12-25T21:12:46.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.245+0000 7fc126828700 1 -- 172.21.15.174:0/1085155941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1200f5410 msgr2=0x7fc120109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:46.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.245+0000 7fc126828700 1 --2- 172.21.15.174:0/1085155941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1200f5410 0x7fc120109400 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fc110013510 tx=0x7fc11000b040).stop 2023-12-25T21:12:46.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.245+0000 7fc126828700 1 -- 172.21.15.174:0/1085155941 shutdown_connections 2023-12-25T21:12:46.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.245+0000 7fc126828700 1 --2- 172.21.15.174:0/1085155941 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fc10802ff60 0x7fc108032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:46.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.245+0000 7fc126828700 1 --2- 172.21.15.174:0/1085155941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1200f5410 0x7fc120109400 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:46.248 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.245+0000 7fc126828700 1 -- 172.21.15.174:0/1085155941 >> 172.21.15.174:0/1085155941 conn(0x7fc1200f0b70 msgr2=0x7fc1200f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:46.248 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.245+0000 7fc126828700 1 -- 172.21.15.174:0/1085155941 shutdown_connections 2023-12-25T21:12:46.249 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:46.245+0000 7fc126828700 1 -- 172.21.15.174:0/1085155941 wait complete. 2023-12-25T21:12:46.249 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:46.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:46 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:44.622148+0000 mgr.smithi061.jxxeke (mgr.14164) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:46.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:46 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:46.243980+0000 mon.smithi061 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.174:0/1085155941' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:47.656 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:47.656 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:48.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:48 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:46.622539+0000 mgr.smithi061.jxxeke (mgr.14164) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:48.832 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.831+0000 7f6774ee2700 1 -- 172.21.15.174:0/3887788942 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67700f5410 msgr2=0x7f67700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:48.832 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.831+0000 7f6774ee2700 1 --2- 172.21.15.174:0/3887788942 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67700f5410 0x7f67700f5830 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f6764009fb0 tx=0x7f6764009b30).stop 2023-12-25T21:12:48.833 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.831+0000 7f6774ee2700 1 -- 172.21.15.174:0/3887788942 shutdown_connections 2023-12-25T21:12:48.833 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.831+0000 7f6774ee2700 1 --2- 172.21.15.174:0/3887788942 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67700f5410 0x7f67700f5830 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:48.833 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.831+0000 7f6774ee2700 1 -- 172.21.15.174:0/3887788942 >> 172.21.15.174:0/3887788942 conn(0x7f67700f0b70 msgr2=0x7f67700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:48.833 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.831+0000 7f6774ee2700 1 -- 172.21.15.174:0/3887788942 shutdown_connections 2023-12-25T21:12:48.833 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.831+0000 7f6774ee2700 1 -- 172.21.15.174:0/3887788942 wait complete. 2023-12-25T21:12:48.833 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.831+0000 7f6774ee2700 1 Processor -- start 2023-12-25T21:12:48.833 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.832+0000 7f6774ee2700 1 -- start start 2023-12-25T21:12:48.833 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.832+0000 7f6774ee2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67700f5410 0x7f6770105640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:48.833 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.832+0000 7f6774ee2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6770105b30 con 0x7f67700f5410 2023-12-25T21:12:48.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.832+0000 7f676e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67700f5410 0x7f6770105640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:48.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.832+0000 7f676e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67700f5410 0x7f6770105640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:41790/0 (socket says 172.21.15.174:41790) 2023-12-25T21:12:48.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.832+0000 7f676e59c700 1 -- 172.21.15.174:0/2316992022 learned_addr learned my addr 172.21.15.174:0/2316992022 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:48.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.832+0000 7f676e59c700 1 -- 172.21.15.174:0/2316992022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6764005040 con 0x7f67700f5410 2023-12-25T21:12:48.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.832+0000 7f676e59c700 1 --2- 172.21.15.174:0/2316992022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67700f5410 0x7f6770105640 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f6764000c00 tx=0x7f676400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:48.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.833+0000 7f675effd700 1 -- 172.21.15.174:0/2316992022 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67640095f0 con 0x7f67700f5410 2023-12-25T21:12:48.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.833+0000 7f6774ee2700 1 -- 172.21.15.174:0/2316992022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6770105d30 con 0x7f67700f5410 2023-12-25T21:12:48.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.833+0000 7f675effd700 1 -- 172.21.15.174:0/2316992022 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6764007e70 con 0x7f67700f5410 2023-12-25T21:12:48.836 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.833+0000 7f675effd700 1 -- 172.21.15.174:0/2316992022 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6764014da0 con 0x7f67700f5410 2023-12-25T21:12:48.836 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.833+0000 7f6774ee2700 1 -- 172.21.15.174:0/2316992022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67701061c0 con 0x7f67700f5410 2023-12-25T21:12:48.836 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.833+0000 7f675effd700 1 -- 172.21.15.174:0/2316992022 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6764025070 con 0x7f67700f5410 2023-12-25T21:12:48.836 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.834+0000 7f6774ee2700 1 -- 172.21.15.174:0/2316992022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6770045ad0 con 0x7f67700f5410 2023-12-25T21:12:48.836 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.834+0000 7f675effd700 1 --2- 172.21.15.174:0/2316992022 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f675802ffb0 0x7f6758032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:48.836 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.834+0000 7f675effd700 1 -- 172.21.15.174:0/2316992022 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f67640408d0 con 0x7f67700f5410 2023-12-25T21:12:48.836 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.834+0000 7f676dd9b700 1 --2- 172.21.15.174:0/2316992022 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f675802ffb0 0x7f6758032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:48.836 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.834+0000 7f676dd9b700 1 --2- 172.21.15.174:0/2316992022 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f675802ffb0 0x7f6758032470 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f67600096a0 tx=0x7f6760006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:48.839 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:48.839+0000 7f675effd700 1 -- 172.21.15.174:0/2316992022 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f676400f4e0 con 0x7f67700f5410 2023-12-25T21:12:49.045 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:49.044+0000 7f6774ee2700 1 -- 172.21.15.174:0/2316992022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6770107070 con 0x7f67700f5410 2023-12-25T21:12:49.045 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:49.045+0000 7f675effd700 1 -- 172.21.15.174:0/2316992022 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f676400f300 con 0x7f67700f5410 2023-12-25T21:12:49.046 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:49.046 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:49.047 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:49.046+0000 7f6774ee2700 1 -- 172.21.15.174:0/2316992022 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f675802ffb0 msgr2=0x7f6758032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:49.047 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:49.046+0000 7f6774ee2700 1 --2- 172.21.15.174:0/2316992022 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f675802ffb0 0x7f6758032470 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f67600096a0 tx=0x7f6760006b40).stop 2023-12-25T21:12:49.047 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:49.046+0000 7f6774ee2700 1 -- 172.21.15.174:0/2316992022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67700f5410 msgr2=0x7f6770105640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:49.048 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:49.046+0000 7f6774ee2700 1 --2- 172.21.15.174:0/2316992022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67700f5410 0x7f6770105640 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f6764000c00 tx=0x7f676400b040).stop 2023-12-25T21:12:49.048 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:49.047+0000 7f6774ee2700 1 -- 172.21.15.174:0/2316992022 shutdown_connections 2023-12-25T21:12:49.048 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:49.047+0000 7f6774ee2700 1 --2- 172.21.15.174:0/2316992022 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f675802ffb0 0x7f6758032470 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:49.048 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:49.047+0000 7f6774ee2700 1 --2- 172.21.15.174:0/2316992022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67700f5410 0x7f6770105640 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:49.048 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:49.047+0000 7f6774ee2700 1 -- 172.21.15.174:0/2316992022 >> 172.21.15.174:0/2316992022 conn(0x7f67700f0b70 msgr2=0x7f67700f1680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:49.048 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:49.047+0000 7f6774ee2700 1 -- 172.21.15.174:0/2316992022 shutdown_connections 2023-12-25T21:12:49.048 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:49.047+0000 7f6774ee2700 1 -- 172.21.15.174:0/2316992022 wait complete. 2023-12-25T21:12:49.048 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:49.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:49 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:49.046405+0000 mon.smithi061 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.174:0/2316992022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:50.445 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:50.446 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:50.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:50 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:48.622774+0000 mgr.smithi061.jxxeke (mgr.14164) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:51.611 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.610+0000 7f40c5026700 1 -- 172.21.15.174:0/2765464569 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40c00f51a0 msgr2=0x7f40c00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:51.611 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.610+0000 7f40c5026700 1 --2- 172.21.15.174:0/2765464569 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40c00f51a0 0x7f40c00f55c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f40b0009fb0 tx=0x7f40b0009b30).stop 2023-12-25T21:12:51.611 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.610+0000 7f40c5026700 1 -- 172.21.15.174:0/2765464569 shutdown_connections 2023-12-25T21:12:51.612 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.610+0000 7f40c5026700 1 --2- 172.21.15.174:0/2765464569 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40c00f51a0 0x7f40c00f55c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:51.612 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.610+0000 7f40c5026700 1 -- 172.21.15.174:0/2765464569 >> 172.21.15.174:0/2765464569 conn(0x7f40c00f09a0 msgr2=0x7f40c00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:51.612 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.610+0000 7f40c5026700 1 -- 172.21.15.174:0/2765464569 shutdown_connections 2023-12-25T21:12:51.612 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.610+0000 7f40c5026700 1 -- 172.21.15.174:0/2765464569 wait complete. 2023-12-25T21:12:51.612 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.610+0000 7f40c5026700 1 Processor -- start 2023-12-25T21:12:51.612 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.610+0000 7f40c5026700 1 -- start start 2023-12-25T21:12:51.612 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.611+0000 7f40c5026700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40c00f51a0 0x7f40c01064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:51.612 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.611+0000 7f40c5026700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c01069b0 con 0x7f40c00f51a0 2023-12-25T21:12:51.614 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.611+0000 7f40bffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40c00f51a0 0x7f40c01064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:51.614 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.611+0000 7f40bffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40c00f51a0 0x7f40c01064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:41802/0 (socket says 172.21.15.174:41802) 2023-12-25T21:12:51.614 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.611+0000 7f40bffff700 1 -- 172.21.15.174:0/883426952 learned_addr learned my addr 172.21.15.174:0/883426952 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:51.614 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.611+0000 7f40bffff700 1 -- 172.21.15.174:0/883426952 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b0005040 con 0x7f40c00f51a0 2023-12-25T21:12:51.614 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.612+0000 7f40bffff700 1 --2- 172.21.15.174:0/883426952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40c00f51a0 0x7f40c01064c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f40b0013510 tx=0x7f40b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:51.614 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.612+0000 7f40bcff9700 1 -- 172.21.15.174:0/883426952 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40b0009320 con 0x7f40c00f51a0 2023-12-25T21:12:51.614 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.612+0000 7f40c5026700 1 -- 172.21.15.174:0/883426952 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c0106bb0 con 0x7f40c00f51a0 2023-12-25T21:12:51.614 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.612+0000 7f40bcff9700 1 -- 172.21.15.174:0/883426952 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40b0013f00 con 0x7f40c00f51a0 2023-12-25T21:12:51.614 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.612+0000 7f40bcff9700 1 -- 172.21.15.174:0/883426952 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40b0014dc0 con 0x7f40c00f51a0 2023-12-25T21:12:51.614 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.612+0000 7f40c5026700 1 -- 172.21.15.174:0/883426952 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c0107040 con 0x7f40c00f51a0 2023-12-25T21:12:51.614 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.613+0000 7f40bcff9700 1 -- 172.21.15.174:0/883426952 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40b00095f0 con 0x7f40c00f51a0 2023-12-25T21:12:51.614 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.613+0000 7f40c5026700 1 -- 172.21.15.174:0/883426952 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c0045ad0 con 0x7f40c00f51a0 2023-12-25T21:12:51.614 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.613+0000 7f40bcff9700 1 --2- 172.21.15.174:0/883426952 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f40a802ff60 0x7f40a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:51.615 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.613+0000 7f40bcff9700 1 -- 172.21.15.174:0/883426952 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f40b0041040 con 0x7f40c00f51a0 2023-12-25T21:12:51.615 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.613+0000 7f40bf7fe700 1 --2- 172.21.15.174:0/883426952 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f40a802ff60 0x7f40a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:51.615 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.613+0000 7f40bf7fe700 1 --2- 172.21.15.174:0/883426952 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f40a802ff60 0x7f40a8032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f40b40096a0 tx=0x7f40b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:51.618 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.618+0000 7f40bcff9700 1 -- 172.21.15.174:0/883426952 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40b0014420 con 0x7f40c00f51a0 2023-12-25T21:12:51.825 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.824+0000 7f40c5026700 1 -- 172.21.15.174:0/883426952 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40c0107ef0 con 0x7f40c00f51a0 2023-12-25T21:12:51.826 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.825+0000 7f40bcff9700 1 -- 172.21.15.174:0/883426952 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f40b0010230 con 0x7f40c00f51a0 2023-12-25T21:12:51.826 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:51.826 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:51.827 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.826+0000 7f40c5026700 1 -- 172.21.15.174:0/883426952 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f40a802ff60 msgr2=0x7f40a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:51.827 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.826+0000 7f40c5026700 1 --2- 172.21.15.174:0/883426952 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f40a802ff60 0x7f40a8032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f40b40096a0 tx=0x7f40b4006b40).stop 2023-12-25T21:12:51.827 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.826+0000 7f40c5026700 1 -- 172.21.15.174:0/883426952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40c00f51a0 msgr2=0x7f40c01064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:51.827 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.826+0000 7f40c5026700 1 --2- 172.21.15.174:0/883426952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40c00f51a0 0x7f40c01064c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f40b0013510 tx=0x7f40b000b040).stop 2023-12-25T21:12:51.827 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.826+0000 7f40c5026700 1 -- 172.21.15.174:0/883426952 shutdown_connections 2023-12-25T21:12:51.828 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.826+0000 7f40c5026700 1 --2- 172.21.15.174:0/883426952 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f40a802ff60 0x7f40a8032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:51.828 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.826+0000 7f40c5026700 1 --2- 172.21.15.174:0/883426952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40c00f51a0 0x7f40c01064c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:51.828 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.826+0000 7f40c5026700 1 -- 172.21.15.174:0/883426952 >> 172.21.15.174:0/883426952 conn(0x7f40c00f09a0 msgr2=0x7f40c00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:51.828 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.827+0000 7f40c5026700 1 -- 172.21.15.174:0/883426952 shutdown_connections 2023-12-25T21:12:51.828 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:51.827+0000 7f40c5026700 1 -- 172.21.15.174:0/883426952 wait complete. 2023-12-25T21:12:51.828 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:52.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:52 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:50.623039+0000 mgr.smithi061.jxxeke (mgr.14164) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:52.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:52 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:51.826395+0000 mon.smithi061 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.174:0/883426952' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:53.227 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:53.227 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:53.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:52.707346+0000 mon.smithi061 (mon.0) 256 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/mirror_snapshot_schedule"}]: dispatch 2023-12-25T21:12:53.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:52.708227+0000 mon.smithi061 (mon.0) 257 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/trash_purge_schedule"}]: dispatch 2023-12-25T21:12:54.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.336+0000 7f8b7507f700 1 -- 172.21.15.174:0/1366074549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b700f5410 msgr2=0x7f8b700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:54.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.336+0000 7f8b7507f700 1 --2- 172.21.15.174:0/1366074549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b700f5410 0x7f8b700f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f8b58009fb0 tx=0x7f8b58009b30).stop 2023-12-25T21:12:54.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.336+0000 7f8b7507f700 1 -- 172.21.15.174:0/1366074549 shutdown_connections 2023-12-25T21:12:54.338 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.336+0000 7f8b7507f700 1 --2- 172.21.15.174:0/1366074549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b700f5410 0x7f8b700f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:54.338 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.336+0000 7f8b7507f700 1 -- 172.21.15.174:0/1366074549 >> 172.21.15.174:0/1366074549 conn(0x7f8b700f0b70 msgr2=0x7f8b700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:54.338 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.336+0000 7f8b7507f700 1 -- 172.21.15.174:0/1366074549 shutdown_connections 2023-12-25T21:12:54.338 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.336+0000 7f8b7507f700 1 -- 172.21.15.174:0/1366074549 wait complete. 2023-12-25T21:12:54.339 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.337+0000 7f8b7507f700 1 Processor -- start 2023-12-25T21:12:54.339 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.337+0000 7f8b7507f700 1 -- start start 2023-12-25T21:12:54.339 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.337+0000 7f8b7507f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b700f5410 0x7f8b70106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:54.339 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.337+0000 7f8b7507f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b70106c10 con 0x7f8b700f5410 2023-12-25T21:12:54.339 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.337+0000 7f8b6ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b700f5410 0x7f8b70106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:54.339 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.337+0000 7f8b6ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b700f5410 0x7f8b70106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:41824/0 (socket says 172.21.15.174:41824) 2023-12-25T21:12:54.339 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.337+0000 7f8b6ed9d700 1 -- 172.21.15.174:0/801444280 learned_addr learned my addr 172.21.15.174:0/801444280 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:54.340 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.338+0000 7f8b6ed9d700 1 -- 172.21.15.174:0/801444280 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b58005040 con 0x7f8b700f5410 2023-12-25T21:12:54.340 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.338+0000 7f8b6ed9d700 1 --2- 172.21.15.174:0/801444280 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b700f5410 0x7f8b70106720 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f8b58013510 tx=0x7f8b5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:54.340 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.338+0000 7f8b677fe700 1 -- 172.21.15.174:0/801444280 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b58009320 con 0x7f8b700f5410 2023-12-25T21:12:54.340 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.338+0000 7f8b7507f700 1 -- 172.21.15.174:0/801444280 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b70106e10 con 0x7f8b700f5410 2023-12-25T21:12:54.340 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.338+0000 7f8b677fe700 1 -- 172.21.15.174:0/801444280 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b58013f00 con 0x7f8b700f5410 2023-12-25T21:12:54.340 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.338+0000 7f8b677fe700 1 -- 172.21.15.174:0/801444280 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b58014d90 con 0x7f8b700f5410 2023-12-25T21:12:54.340 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.338+0000 7f8b7507f700 1 -- 172.21.15.174:0/801444280 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b701072a0 con 0x7f8b700f5410 2023-12-25T21:12:54.341 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.339+0000 7f8b677fe700 1 -- 172.21.15.174:0/801444280 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8b58014420 con 0x7f8b700f5410 2023-12-25T21:12:54.341 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.339+0000 7f8b7507f700 1 -- 172.21.15.174:0/801444280 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b70045ad0 con 0x7f8b700f5410 2023-12-25T21:12:54.341 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.339+0000 7f8b677fe700 1 --2- 172.21.15.174:0/801444280 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8b5c02ff60 0x7f8b5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:54.341 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.339+0000 7f8b677fe700 1 -- 172.21.15.174:0/801444280 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8b580410c0 con 0x7f8b700f5410 2023-12-25T21:12:54.341 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.339+0000 7f8b6e59c700 1 --2- 172.21.15.174:0/801444280 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8b5c02ff60 0x7f8b5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:54.341 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.340+0000 7f8b6e59c700 1 --2- 172.21.15.174:0/801444280 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8b5c02ff60 0x7f8b5c032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f8b600096a0 tx=0x7f8b60006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:54.344 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.344+0000 7f8b677fe700 1 -- 172.21.15.174:0/801444280 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b580095f0 con 0x7f8b700f5410 2023-12-25T21:12:54.551 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.549+0000 7f8b7507f700 1 -- 172.21.15.174:0/801444280 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b70108150 con 0x7f8b700f5410 2023-12-25T21:12:54.551 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.550+0000 7f8b677fe700 1 -- 172.21.15.174:0/801444280 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8b5801db70 con 0x7f8b700f5410 2023-12-25T21:12:54.551 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:54.552 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:54.553 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.552+0000 7f8b7507f700 1 -- 172.21.15.174:0/801444280 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8b5c02ff60 msgr2=0x7f8b5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:54.553 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.552+0000 7f8b7507f700 1 --2- 172.21.15.174:0/801444280 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8b5c02ff60 0x7f8b5c032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f8b600096a0 tx=0x7f8b60006b40).stop 2023-12-25T21:12:54.553 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.552+0000 7f8b7507f700 1 -- 172.21.15.174:0/801444280 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b700f5410 msgr2=0x7f8b70106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:54.553 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.552+0000 7f8b7507f700 1 --2- 172.21.15.174:0/801444280 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b700f5410 0x7f8b70106720 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f8b58013510 tx=0x7f8b5800b040).stop 2023-12-25T21:12:54.553 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.552+0000 7f8b7507f700 1 -- 172.21.15.174:0/801444280 shutdown_connections 2023-12-25T21:12:54.553 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.552+0000 7f8b7507f700 1 --2- 172.21.15.174:0/801444280 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8b5c02ff60 0x7f8b5c032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:54.553 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.552+0000 7f8b7507f700 1 --2- 172.21.15.174:0/801444280 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b700f5410 0x7f8b70106720 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:54.553 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.552+0000 7f8b7507f700 1 -- 172.21.15.174:0/801444280 >> 172.21.15.174:0/801444280 conn(0x7f8b700f0b70 msgr2=0x7f8b7005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:54.554 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.552+0000 7f8b7507f700 1 -- 172.21.15.174:0/801444280 shutdown_connections 2023-12-25T21:12:54.554 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:54.552+0000 7f8b7507f700 1 -- 172.21.15.174:0/801444280 wait complete. 2023-12-25T21:12:54.554 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:54.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:54 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:52.623285+0000 mgr.smithi061.jxxeke (mgr.14164) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:55.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:55 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:54.551752+0000 mon.smithi061 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.174:0/801444280' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:55.952 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:55.952 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:56.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:56 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:54.623521+0000 mgr.smithi061.jxxeke (mgr.14164) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:57.139 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.137+0000 7f651b4f3700 1 -- 172.21.15.174:0/3444637505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65140f51a0 msgr2=0x7f65140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:57.139 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.137+0000 7f651b4f3700 1 --2- 172.21.15.174:0/3444637505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65140f51a0 0x7f65140f55c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f6510009fb0 tx=0x7f6510009b30).stop 2023-12-25T21:12:57.139 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.137+0000 7f651b4f3700 1 -- 172.21.15.174:0/3444637505 shutdown_connections 2023-12-25T21:12:57.139 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.137+0000 7f651b4f3700 1 --2- 172.21.15.174:0/3444637505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65140f51a0 0x7f65140f55c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:57.139 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.137+0000 7f651b4f3700 1 -- 172.21.15.174:0/3444637505 >> 172.21.15.174:0/3444637505 conn(0x7f65140f09a0 msgr2=0x7f65140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:57.140 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.137+0000 7f651b4f3700 1 -- 172.21.15.174:0/3444637505 shutdown_connections 2023-12-25T21:12:57.140 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.137+0000 7f651b4f3700 1 -- 172.21.15.174:0/3444637505 wait complete. 2023-12-25T21:12:57.140 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.137+0000 7f651b4f3700 1 Processor -- start 2023-12-25T21:12:57.140 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.137+0000 7f651b4f3700 1 -- start start 2023-12-25T21:12:57.140 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.137+0000 7f651b4f3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65140f51a0 0x7f65141091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:57.140 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.137+0000 7f651b4f3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6514060220 con 0x7f65140f51a0 2023-12-25T21:12:57.140 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.138+0000 7f651a4f1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65140f51a0 0x7f65141091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:57.140 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.138+0000 7f651a4f1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65140f51a0 0x7f65141091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:33742/0 (socket says 172.21.15.174:33742) 2023-12-25T21:12:57.140 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.138+0000 7f651a4f1700 1 -- 172.21.15.174:0/3791978961 learned_addr learned my addr 172.21.15.174:0/3791978961 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:57.140 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.139+0000 7f651a4f1700 1 -- 172.21.15.174:0/3791978961 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6510005040 con 0x7f65140f51a0 2023-12-25T21:12:57.141 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.139+0000 7f651a4f1700 1 --2- 172.21.15.174:0/3791978961 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65140f51a0 0x7f65141091d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f6510013510 tx=0x7f651000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:57.141 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.139+0000 7f650affd700 1 -- 172.21.15.174:0/3791978961 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6510009320 con 0x7f65140f51a0 2023-12-25T21:12:57.142 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.139+0000 7f650affd700 1 -- 172.21.15.174:0/3791978961 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6510013f00 con 0x7f65140f51a0 2023-12-25T21:12:57.142 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.139+0000 7f651b4f3700 1 -- 172.21.15.174:0/3791978961 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6514060420 con 0x7f65140f51a0 2023-12-25T21:12:57.142 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.139+0000 7f650affd700 1 -- 172.21.15.174:0/3791978961 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6510014dc0 con 0x7f65140f51a0 2023-12-25T21:12:57.142 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.139+0000 7f651b4f3700 1 -- 172.21.15.174:0/3791978961 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6514109880 con 0x7f65140f51a0 2023-12-25T21:12:57.142 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.140+0000 7f651b4f3700 1 -- 172.21.15.174:0/3791978961 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6514059100 con 0x7f65140f51a0 2023-12-25T21:12:57.142 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.140+0000 7f650affd700 1 -- 172.21.15.174:0/3791978961 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f65100095f0 con 0x7f65140f51a0 2023-12-25T21:12:57.142 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.140+0000 7f650affd700 1 --2- 172.21.15.174:0/3791978961 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f650002ff60 0x7f6500032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:57.142 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.140+0000 7f650affd700 1 -- 172.21.15.174:0/3791978961 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6510041000 con 0x7f65140f51a0 2023-12-25T21:12:57.142 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.140+0000 7f6519cf0700 1 --2- 172.21.15.174:0/3791978961 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f650002ff60 0x7f6500032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:57.142 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.141+0000 7f6519cf0700 1 --2- 172.21.15.174:0/3791978961 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f650002ff60 0x7f6500032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f65040096a0 tx=0x7f6504006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:57.146 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.145+0000 7f650affd700 1 -- 172.21.15.174:0/3791978961 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6510014420 con 0x7f65140f51a0 2023-12-25T21:12:57.350 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.349+0000 7f651b4f3700 1 -- 172.21.15.174:0/3791978961 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6514045ad0 con 0x7f65140f51a0 2023-12-25T21:12:57.351 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.350+0000 7f650affd700 1 -- 172.21.15.174:0/3791978961 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6510010230 con 0x7f65140f51a0 2023-12-25T21:12:57.351 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:12:57.351 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:12:57.352 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.352+0000 7f651b4f3700 1 -- 172.21.15.174:0/3791978961 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f650002ff60 msgr2=0x7f6500032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:57.353 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.352+0000 7f651b4f3700 1 --2- 172.21.15.174:0/3791978961 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f650002ff60 0x7f6500032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f65040096a0 tx=0x7f6504006b40).stop 2023-12-25T21:12:57.353 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.352+0000 7f651b4f3700 1 -- 172.21.15.174:0/3791978961 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65140f51a0 msgr2=0x7f65141091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:57.353 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.352+0000 7f651b4f3700 1 --2- 172.21.15.174:0/3791978961 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65140f51a0 0x7f65141091d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f6510013510 tx=0x7f651000b040).stop 2023-12-25T21:12:57.353 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.352+0000 7f651b4f3700 1 -- 172.21.15.174:0/3791978961 shutdown_connections 2023-12-25T21:12:57.353 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.352+0000 7f651b4f3700 1 --2- 172.21.15.174:0/3791978961 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f650002ff60 0x7f6500032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:57.353 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.352+0000 7f651b4f3700 1 --2- 172.21.15.174:0/3791978961 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65140f51a0 0x7f65141091d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:57.353 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.352+0000 7f651b4f3700 1 -- 172.21.15.174:0/3791978961 >> 172.21.15.174:0/3791978961 conn(0x7f65140f09a0 msgr2=0x7f65140f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:57.353 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.352+0000 7f651b4f3700 1 -- 172.21.15.174:0/3791978961 shutdown_connections 2023-12-25T21:12:57.353 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:57.352+0000 7f651b4f3700 1 -- 172.21.15.174:0/3791978961 wait complete. 2023-12-25T21:12:57.354 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:12:57.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:57 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:12:57.351650+0000 mon.smithi061 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.174:0/3791978961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:12:58.687 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:12:58.688 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:12:58.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:58 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:56.623758+0000 mgr.smithi061.jxxeke (mgr.14164) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:59.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:12:59 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:12:58.624119+0000 mgr.smithi061.jxxeke (mgr.14164) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:12:59.801 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.799+0000 7f86493d4700 1 -- 172.21.15.174:0/3199357865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86440f4880 msgr2=0x7f86440f4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:12:59.801 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.799+0000 7f86493d4700 1 --2- 172.21.15.174:0/3199357865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86440f4880 0x7f86440f4ca0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f8638009fb0 tx=0x7f8638009b30).stop 2023-12-25T21:12:59.801 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.799+0000 7f86493d4700 1 -- 172.21.15.174:0/3199357865 shutdown_connections 2023-12-25T21:12:59.801 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.799+0000 7f86493d4700 1 --2- 172.21.15.174:0/3199357865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86440f4880 0x7f86440f4ca0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:12:59.801 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.799+0000 7f86493d4700 1 -- 172.21.15.174:0/3199357865 >> 172.21.15.174:0/3199357865 conn(0x7f86440f02e0 msgr2=0x7f86440f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:12:59.801 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.799+0000 7f86493d4700 1 -- 172.21.15.174:0/3199357865 shutdown_connections 2023-12-25T21:12:59.801 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.799+0000 7f86493d4700 1 -- 172.21.15.174:0/3199357865 wait complete. 2023-12-25T21:12:59.801 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.800+0000 7f86493d4700 1 Processor -- start 2023-12-25T21:12:59.802 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.800+0000 7f86493d4700 1 -- start start 2023-12-25T21:12:59.802 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.800+0000 7f86493d4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86440f4880 0x7f86441066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:59.802 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.800+0000 7f86493d4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8644106ba0 con 0x7f86440f4880 2023-12-25T21:12:59.802 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.800+0000 7f8642ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86440f4880 0x7f86441066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:59.802 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.800+0000 7f8642ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86440f4880 0x7f86441066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:33764/0 (socket says 172.21.15.174:33764) 2023-12-25T21:12:59.802 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.800+0000 7f8642ffd700 1 -- 172.21.15.174:0/1861667935 learned_addr learned my addr 172.21.15.174:0/1861667935 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:12:59.803 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.801+0000 7f8642ffd700 1 -- 172.21.15.174:0/1861667935 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8638005040 con 0x7f86440f4880 2023-12-25T21:12:59.803 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.801+0000 7f8642ffd700 1 --2- 172.21.15.174:0/1861667935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86440f4880 0x7f86441066b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f8638013510 tx=0x7f863800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:59.803 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.801+0000 7f862b7fe700 1 -- 172.21.15.174:0/1861667935 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8638009320 con 0x7f86440f4880 2023-12-25T21:12:59.804 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.801+0000 7f86493d4700 1 -- 172.21.15.174:0/1861667935 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8644106da0 con 0x7f86440f4880 2023-12-25T21:12:59.804 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.801+0000 7f862b7fe700 1 -- 172.21.15.174:0/1861667935 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8638013f00 con 0x7f86440f4880 2023-12-25T21:12:59.804 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.801+0000 7f862b7fe700 1 -- 172.21.15.174:0/1861667935 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8638014dc0 con 0x7f86440f4880 2023-12-25T21:12:59.804 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.801+0000 7f86493d4700 1 -- 172.21.15.174:0/1861667935 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8644107230 con 0x7f86440f4880 2023-12-25T21:12:59.804 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.802+0000 7f862b7fe700 1 -- 172.21.15.174:0/1861667935 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86380095f0 con 0x7f86440f4880 2023-12-25T21:12:59.804 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.802+0000 7f86493d4700 1 -- 172.21.15.174:0/1861667935 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8644045ad0 con 0x7f86440f4880 2023-12-25T21:12:59.804 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.802+0000 7f862b7fe700 1 --2- 172.21.15.174:0/1861667935 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f862c02ff60 0x7f862c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:12:59.804 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.802+0000 7f862b7fe700 1 -- 172.21.15.174:0/1861667935 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f86380410f0 con 0x7f86440f4880 2023-12-25T21:12:59.804 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.802+0000 7f86427fc700 1 --2- 172.21.15.174:0/1861667935 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f862c02ff60 0x7f862c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:12:59.804 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.803+0000 7f86427fc700 1 --2- 172.21.15.174:0/1861667935 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f862c02ff60 0x7f862c032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f86340096a0 tx=0x7f8634006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:12:59.807 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:12:59.807+0000 7f862b7fe700 1 -- 172.21.15.174:0/1861667935 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8638014420 con 0x7f86440f4880 2023-12-25T21:13:00.015 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:00.014+0000 7f86493d4700 1 -- 172.21.15.174:0/1861667935 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f86440fbf50 con 0x7f86440f4880 2023-12-25T21:13:00.016 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:00.015+0000 7f862b7fe700 1 -- 172.21.15.174:0/1861667935 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8638010230 con 0x7f86440f4880 2023-12-25T21:13:00.016 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:00.016 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:00.017 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:00.016+0000 7f86493d4700 1 -- 172.21.15.174:0/1861667935 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f862c02ff60 msgr2=0x7f862c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:00.017 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:00.016+0000 7f86493d4700 1 --2- 172.21.15.174:0/1861667935 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f862c02ff60 0x7f862c032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f86340096a0 tx=0x7f8634006b40).stop 2023-12-25T21:13:00.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:00.016+0000 7f86493d4700 1 -- 172.21.15.174:0/1861667935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86440f4880 msgr2=0x7f86441066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:00.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:00.016+0000 7f86493d4700 1 --2- 172.21.15.174:0/1861667935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86440f4880 0x7f86441066b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f8638013510 tx=0x7f863800b040).stop 2023-12-25T21:13:00.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:00.017+0000 7f86493d4700 1 -- 172.21.15.174:0/1861667935 shutdown_connections 2023-12-25T21:13:00.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:00.017+0000 7f86493d4700 1 --2- 172.21.15.174:0/1861667935 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f862c02ff60 0x7f862c032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:00.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:00.017+0000 7f86493d4700 1 --2- 172.21.15.174:0/1861667935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86440f4880 0x7f86441066b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:00.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:00.017+0000 7f86493d4700 1 -- 172.21.15.174:0/1861667935 >> 172.21.15.174:0/1861667935 conn(0x7f86440f02e0 msgr2=0x7f86440f0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:00.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:00.017+0000 7f86493d4700 1 -- 172.21.15.174:0/1861667935 shutdown_connections 2023-12-25T21:13:00.018 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:00.017+0000 7f86493d4700 1 -- 172.21.15.174:0/1861667935 wait complete. 2023-12-25T21:13:00.018 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:00.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:00 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:00.016130+0000 mon.smithi061 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.174:0/1861667935' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:01.396 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:01.396 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:02.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.544+0000 7f36732d5700 1 -- 172.21.15.174:0/3908296091 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c060280 msgr2=0x7f366c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:02.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.544+0000 7f36732d5700 1 --2- 172.21.15.174:0/3908296091 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c060280 0x7f366c0f90f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f365c009fb0 tx=0x7f365c009b30).stop 2023-12-25T21:13:02.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.544+0000 7f36732d5700 1 -- 172.21.15.174:0/3908296091 shutdown_connections 2023-12-25T21:13:02.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.544+0000 7f36732d5700 1 --2- 172.21.15.174:0/3908296091 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c060280 0x7f366c0f90f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:02.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.544+0000 7f36732d5700 1 -- 172.21.15.174:0/3908296091 >> 172.21.15.174:0/3908296091 conn(0x7f366c0f0b70 msgr2=0x7f366c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:02.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.544+0000 7f36732d5700 1 -- 172.21.15.174:0/3908296091 shutdown_connections 2023-12-25T21:13:02.548 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.544+0000 7f36732d5700 1 -- 172.21.15.174:0/3908296091 wait complete. 2023-12-25T21:13:02.548 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.544+0000 7f36732d5700 1 Processor -- start 2023-12-25T21:13:02.548 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.544+0000 7f36732d5700 1 -- start start 2023-12-25T21:13:02.548 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.544+0000 7f36732d5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c060280 0x7f366c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:02.548 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.544+0000 7f36732d5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c106bd0 con 0x7f366c060280 2023-12-25T21:13:02.548 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.545+0000 7f3671071700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c060280 0x7f366c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:02.548 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.545+0000 7f3671071700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c060280 0x7f366c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:33774/0 (socket says 172.21.15.174:33774) 2023-12-25T21:13:02.548 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.545+0000 7f3671071700 1 -- 172.21.15.174:0/4260518975 learned_addr learned my addr 172.21.15.174:0/4260518975 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:02.548 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.545+0000 7f3671071700 1 -- 172.21.15.174:0/4260518975 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365c005040 con 0x7f366c060280 2023-12-25T21:13:02.548 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.545+0000 7f3671071700 1 --2- 172.21.15.174:0/4260518975 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c060280 0x7f366c1066e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f365c013510 tx=0x7f365c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:02.548 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.545+0000 7f3661ffb700 1 -- 172.21.15.174:0/4260518975 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f365c009320 con 0x7f366c060280 2023-12-25T21:13:02.548 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.545+0000 7f36732d5700 1 -- 172.21.15.174:0/4260518975 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f366c106dd0 con 0x7f366c060280 2023-12-25T21:13:02.548 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.545+0000 7f3661ffb700 1 -- 172.21.15.174:0/4260518975 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f365c013f00 con 0x7f366c060280 2023-12-25T21:13:02.548 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.545+0000 7f3661ffb700 1 -- 172.21.15.174:0/4260518975 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f365c014dc0 con 0x7f366c060280 2023-12-25T21:13:02.549 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.546+0000 7f36732d5700 1 -- 172.21.15.174:0/4260518975 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366c107260 con 0x7f366c060280 2023-12-25T21:13:02.549 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.547+0000 7f3661ffb700 1 -- 172.21.15.174:0/4260518975 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f365c0095f0 con 0x7f366c060280 2023-12-25T21:13:02.550 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.547+0000 7f36732d5700 1 -- 172.21.15.174:0/4260518975 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f366c045ad0 con 0x7f366c060280 2023-12-25T21:13:02.550 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.547+0000 7f3661ffb700 1 --2- 172.21.15.174:0/4260518975 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f365802ff60 0x7f3658032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:02.550 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.547+0000 7f3661ffb700 1 -- 172.21.15.174:0/4260518975 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f365c041040 con 0x7f366c060280 2023-12-25T21:13:02.550 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.547+0000 7f3670870700 1 --2- 172.21.15.174:0/4260518975 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f365802ff60 0x7f3658032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:02.550 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.547+0000 7f3670870700 1 --2- 172.21.15.174:0/4260518975 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f365802ff60 0x7f3658032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f36680096a0 tx=0x7f3668006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:02.552 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.552+0000 7f3661ffb700 1 -- 172.21.15.174:0/4260518975 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f365c014420 con 0x7f366c060280 2023-12-25T21:13:02.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:02 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:00.624397+0000 mgr.smithi061.jxxeke (mgr.14164) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:02.759 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.758+0000 7f36732d5700 1 -- 172.21.15.174:0/4260518975 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f366c108110 con 0x7f366c060280 2023-12-25T21:13:02.760 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.759+0000 7f3661ffb700 1 -- 172.21.15.174:0/4260518975 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f365c010230 con 0x7f366c060280 2023-12-25T21:13:02.760 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:02.760 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:02.762 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.761+0000 7f36732d5700 1 -- 172.21.15.174:0/4260518975 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f365802ff60 msgr2=0x7f3658032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:02.762 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.761+0000 7f36732d5700 1 --2- 172.21.15.174:0/4260518975 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f365802ff60 0x7f3658032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f36680096a0 tx=0x7f3668006b40).stop 2023-12-25T21:13:02.762 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.761+0000 7f36732d5700 1 -- 172.21.15.174:0/4260518975 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c060280 msgr2=0x7f366c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:02.762 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.761+0000 7f36732d5700 1 --2- 172.21.15.174:0/4260518975 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c060280 0x7f366c1066e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f365c013510 tx=0x7f365c00b040).stop 2023-12-25T21:13:02.762 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.761+0000 7f36732d5700 1 -- 172.21.15.174:0/4260518975 shutdown_connections 2023-12-25T21:13:02.762 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.761+0000 7f36732d5700 1 --2- 172.21.15.174:0/4260518975 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f365802ff60 0x7f3658032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:02.762 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.761+0000 7f36732d5700 1 --2- 172.21.15.174:0/4260518975 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f366c060280 0x7f366c1066e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:02.762 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.761+0000 7f36732d5700 1 -- 172.21.15.174:0/4260518975 >> 172.21.15.174:0/4260518975 conn(0x7f366c0f0b70 msgr2=0x7f366c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:02.763 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.761+0000 7f36732d5700 1 -- 172.21.15.174:0/4260518975 shutdown_connections 2023-12-25T21:13:02.763 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:02.761+0000 7f36732d5700 1 -- 172.21.15.174:0/4260518975 wait complete. 2023-12-25T21:13:02.763 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:03.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:02.760632+0000 mon.smithi061 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.174:0/4260518975' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:04.142 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:04.142 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:04.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:04 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:02.624743+0000 mgr.smithi061.jxxeke (mgr.14164) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:05.372 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.370+0000 7f20801b1700 1 -- 172.21.15.174:0/1492076893 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5410 msgr2=0x7f20780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:05.373 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.370+0000 7f20801b1700 1 --2- 172.21.15.174:0/1492076893 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5410 0x7f20780f5830 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f2068009fb0 tx=0x7f2068009b30).stop 2023-12-25T21:13:05.373 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.370+0000 7f20801b1700 1 -- 172.21.15.174:0/1492076893 shutdown_connections 2023-12-25T21:13:05.373 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.370+0000 7f20801b1700 1 --2- 172.21.15.174:0/1492076893 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5410 0x7f20780f5830 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:05.373 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.370+0000 7f20801b1700 1 -- 172.21.15.174:0/1492076893 >> 172.21.15.174:0/1492076893 conn(0x7f20780f0b70 msgr2=0x7f20780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:05.373 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.370+0000 7f20801b1700 1 -- 172.21.15.174:0/1492076893 shutdown_connections 2023-12-25T21:13:05.373 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.370+0000 7f20801b1700 1 -- 172.21.15.174:0/1492076893 wait complete. 2023-12-25T21:13:05.373 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.371+0000 7f20801b1700 1 Processor -- start 2023-12-25T21:13:05.373 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.371+0000 7f20801b1700 1 -- start start 2023-12-25T21:13:05.373 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.371+0000 7f20801b1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5410 0x7f2078105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:05.373 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.371+0000 7f20801b1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2078105b20 con 0x7f20780f5410 2023-12-25T21:13:05.374 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.371+0000 7f207df4d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5410 0x7f2078105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:05.374 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.371+0000 7f207df4d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5410 0x7f2078105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:33798/0 (socket says 172.21.15.174:33798) 2023-12-25T21:13:05.374 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.371+0000 7f207df4d700 1 -- 172.21.15.174:0/3547986441 learned_addr learned my addr 172.21.15.174:0/3547986441 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:05.374 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.371+0000 7f207df4d700 1 -- 172.21.15.174:0/3547986441 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2068005040 con 0x7f20780f5410 2023-12-25T21:13:05.374 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.371+0000 7f207df4d700 1 --2- 172.21.15.174:0/3547986441 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5410 0x7f2078105630 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f2068013510 tx=0x7f206800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:05.374 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.372+0000 7f206e7fc700 1 -- 172.21.15.174:0/3547986441 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2068009320 con 0x7f20780f5410 2023-12-25T21:13:05.374 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.372+0000 7f20801b1700 1 -- 172.21.15.174:0/3547986441 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2078105d20 con 0x7f20780f5410 2023-12-25T21:13:05.374 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.372+0000 7f206e7fc700 1 -- 172.21.15.174:0/3547986441 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2068013f00 con 0x7f20780f5410 2023-12-25T21:13:05.374 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.372+0000 7f206e7fc700 1 -- 172.21.15.174:0/3547986441 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2068014d90 con 0x7f20780f5410 2023-12-25T21:13:05.374 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.372+0000 7f20801b1700 1 -- 172.21.15.174:0/3547986441 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20781061b0 con 0x7f20780f5410 2023-12-25T21:13:05.375 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.372+0000 7f20801b1700 1 -- 172.21.15.174:0/3547986441 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2078045ad0 con 0x7f20780f5410 2023-12-25T21:13:05.375 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.373+0000 7f206e7fc700 1 -- 172.21.15.174:0/3547986441 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f20680095f0 con 0x7f20780f5410 2023-12-25T21:13:05.375 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.373+0000 7f206e7fc700 1 --2- 172.21.15.174:0/3547986441 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f206402ff60 0x7f2064032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:05.375 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.373+0000 7f206e7fc700 1 -- 172.21.15.174:0/3547986441 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2068040fe0 con 0x7f20780f5410 2023-12-25T21:13:05.375 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.374+0000 7f207d74c700 1 --2- 172.21.15.174:0/3547986441 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f206402ff60 0x7f2064032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:05.376 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.375+0000 7f207d74c700 1 --2- 172.21.15.174:0/3547986441 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f206402ff60 0x7f2064032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f20740096a0 tx=0x7f2074006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:05.378 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.377+0000 7f206e7fc700 1 -- 172.21.15.174:0/3547986441 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f206801d7f0 con 0x7f20780f5410 2023-12-25T21:13:05.585 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.584+0000 7f20801b1700 1 -- 172.21.15.174:0/3547986441 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20780fae60 con 0x7f20780f5410 2023-12-25T21:13:05.585 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.585+0000 7f206e7fc700 1 -- 172.21.15.174:0/3547986441 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2068010230 con 0x7f20780f5410 2023-12-25T21:13:05.586 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:05.586 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:05.587 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.586+0000 7f20801b1700 1 -- 172.21.15.174:0/3547986441 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f206402ff60 msgr2=0x7f2064032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:05.587 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.586+0000 7f20801b1700 1 --2- 172.21.15.174:0/3547986441 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f206402ff60 0x7f2064032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f20740096a0 tx=0x7f2074006b40).stop 2023-12-25T21:13:05.588 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.586+0000 7f20801b1700 1 -- 172.21.15.174:0/3547986441 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5410 msgr2=0x7f2078105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:05.588 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.586+0000 7f20801b1700 1 --2- 172.21.15.174:0/3547986441 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5410 0x7f2078105630 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f2068013510 tx=0x7f206800b040).stop 2023-12-25T21:13:05.588 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.586+0000 7f20801b1700 1 -- 172.21.15.174:0/3547986441 shutdown_connections 2023-12-25T21:13:05.588 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.586+0000 7f20801b1700 1 --2- 172.21.15.174:0/3547986441 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f206402ff60 0x7f2064032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:05.588 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.586+0000 7f20801b1700 1 --2- 172.21.15.174:0/3547986441 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20780f5410 0x7f2078105630 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:05.588 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.586+0000 7f20801b1700 1 -- 172.21.15.174:0/3547986441 >> 172.21.15.174:0/3547986441 conn(0x7f20780f0b70 msgr2=0x7f20780f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:05.588 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.586+0000 7f20801b1700 1 -- 172.21.15.174:0/3547986441 shutdown_connections 2023-12-25T21:13:05.588 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:05.586+0000 7f20801b1700 1 -- 172.21.15.174:0/3547986441 wait complete. 2023-12-25T21:13:05.588 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:06.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:06 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:04.625130+0000 mgr.smithi061.jxxeke (mgr.14164) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:06.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:06 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:05.586282+0000 mon.smithi061 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.174:0/3547986441' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:06.975 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:06.976 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:08.084 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.081+0000 7f5f7464e700 1 -- 172.21.15.174:0/2474023824 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f6c0f5410 msgr2=0x7f5f6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:08.084 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.081+0000 7f5f7464e700 1 --2- 172.21.15.174:0/2474023824 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f6c0f5410 0x7f5f6c0f5830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f5f5c009fb0 tx=0x7f5f5c009b30).stop 2023-12-25T21:13:08.084 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.081+0000 7f5f7464e700 1 -- 172.21.15.174:0/2474023824 shutdown_connections 2023-12-25T21:13:08.084 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.081+0000 7f5f7464e700 1 --2- 172.21.15.174:0/2474023824 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f6c0f5410 0x7f5f6c0f5830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:08.084 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.081+0000 7f5f7464e700 1 -- 172.21.15.174:0/2474023824 >> 172.21.15.174:0/2474023824 conn(0x7f5f6c0f0b70 msgr2=0x7f5f6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:08.084 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.081+0000 7f5f7464e700 1 -- 172.21.15.174:0/2474023824 shutdown_connections 2023-12-25T21:13:08.084 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.081+0000 7f5f7464e700 1 -- 172.21.15.174:0/2474023824 wait complete. 2023-12-25T21:13:08.084 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.082+0000 7f5f7464e700 1 Processor -- start 2023-12-25T21:13:08.084 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.082+0000 7f5f7464e700 1 -- start start 2023-12-25T21:13:08.084 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.082+0000 7f5f7464e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f6c0f5410 0x7f5f6c1055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:08.084 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.082+0000 7f5f7464e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c105ac0 con 0x7f5f6c0f5410 2023-12-25T21:13:08.085 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.082+0000 7f5f723ea700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f6c0f5410 0x7f5f6c1055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:08.085 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.082+0000 7f5f723ea700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f6c0f5410 0x7f5f6c1055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:53622/0 (socket says 172.21.15.174:53622) 2023-12-25T21:13:08.085 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.082+0000 7f5f723ea700 1 -- 172.21.15.174:0/1593932932 learned_addr learned my addr 172.21.15.174:0/1593932932 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:08.085 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.082+0000 7f5f723ea700 1 -- 172.21.15.174:0/1593932932 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5c005040 con 0x7f5f6c0f5410 2023-12-25T21:13:08.085 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.083+0000 7f5f723ea700 1 --2- 172.21.15.174:0/1593932932 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f6c0f5410 0x7f5f6c1055d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f5f5c013510 tx=0x7f5f5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:08.085 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.083+0000 7f5f62ffd700 1 -- 172.21.15.174:0/1593932932 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f5c009320 con 0x7f5f6c0f5410 2023-12-25T21:13:08.085 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.083+0000 7f5f7464e700 1 -- 172.21.15.174:0/1593932932 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f6c105cc0 con 0x7f5f6c0f5410 2023-12-25T21:13:08.085 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.083+0000 7f5f62ffd700 1 -- 172.21.15.174:0/1593932932 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f5c013f00 con 0x7f5f6c0f5410 2023-12-25T21:13:08.085 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.083+0000 7f5f62ffd700 1 -- 172.21.15.174:0/1593932932 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f5c014d90 con 0x7f5f6c0f5410 2023-12-25T21:13:08.085 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.083+0000 7f5f7464e700 1 -- 172.21.15.174:0/1593932932 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f6c106150 con 0x7f5f6c0f5410 2023-12-25T21:13:08.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.084+0000 7f5f62ffd700 1 -- 172.21.15.174:0/1593932932 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5f5c0095f0 con 0x7f5f6c0f5410 2023-12-25T21:13:08.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.084+0000 7f5f7464e700 1 -- 172.21.15.174:0/1593932932 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f6c045ad0 con 0x7f5f6c0f5410 2023-12-25T21:13:08.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.084+0000 7f5f62ffd700 1 --2- 172.21.15.174:0/1593932932 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5f5802ff60 0x7f5f58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:08.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.084+0000 7f5f62ffd700 1 -- 172.21.15.174:0/1593932932 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5f5c0410c0 con 0x7f5f6c0f5410 2023-12-25T21:13:08.086 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.084+0000 7f5f71be9700 1 --2- 172.21.15.174:0/1593932932 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5f5802ff60 0x7f5f58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:08.087 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.085+0000 7f5f71be9700 1 --2- 172.21.15.174:0/1593932932 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5f5802ff60 0x7f5f58032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5f680096a0 tx=0x7f5f68006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:08.090 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.089+0000 7f5f62ffd700 1 -- 172.21.15.174:0/1593932932 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f5c014420 con 0x7f5f6c0f5410 2023-12-25T21:13:08.296 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.295+0000 7f5f7464e700 1 -- 172.21.15.174:0/1593932932 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f6c107000 con 0x7f5f6c0f5410 2023-12-25T21:13:08.296 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.296+0000 7f5f62ffd700 1 -- 172.21.15.174:0/1593932932 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5f5c010230 con 0x7f5f6c0f5410 2023-12-25T21:13:08.297 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:08.297 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:08.298 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.297+0000 7f5f7464e700 1 -- 172.21.15.174:0/1593932932 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5f5802ff60 msgr2=0x7f5f58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:08.298 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.297+0000 7f5f7464e700 1 --2- 172.21.15.174:0/1593932932 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5f5802ff60 0x7f5f58032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5f680096a0 tx=0x7f5f68006b40).stop 2023-12-25T21:13:08.298 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.297+0000 7f5f7464e700 1 -- 172.21.15.174:0/1593932932 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f6c0f5410 msgr2=0x7f5f6c1055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:08.298 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.297+0000 7f5f7464e700 1 --2- 172.21.15.174:0/1593932932 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f6c0f5410 0x7f5f6c1055d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f5f5c013510 tx=0x7f5f5c00b040).stop 2023-12-25T21:13:08.298 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.297+0000 7f5f7464e700 1 -- 172.21.15.174:0/1593932932 shutdown_connections 2023-12-25T21:13:08.298 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.297+0000 7f5f7464e700 1 --2- 172.21.15.174:0/1593932932 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f5f5802ff60 0x7f5f58032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:08.298 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.297+0000 7f5f7464e700 1 --2- 172.21.15.174:0/1593932932 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f6c0f5410 0x7f5f6c1055d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:08.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.297+0000 7f5f7464e700 1 -- 172.21.15.174:0/1593932932 >> 172.21.15.174:0/1593932932 conn(0x7f5f6c0f0b70 msgr2=0x7f5f6c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:08.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.297+0000 7f5f7464e700 1 -- 172.21.15.174:0/1593932932 shutdown_connections 2023-12-25T21:13:08.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:08.297+0000 7f5f7464e700 1 -- 172.21.15.174:0/1593932932 wait complete. 2023-12-25T21:13:08.299 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:08.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:08 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:06.625495+0000 mgr.smithi061.jxxeke (mgr.14164) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:08.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:08 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:08.297122+0000 mon.smithi061 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.174:0/1593932932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:09.744 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:09.744 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:10.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:10 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:08.625870+0000 mgr.smithi061.jxxeke (mgr.14164) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:10.915 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.913+0000 7fbe7600f700 1 -- 172.21.15.174:0/3351931018 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe700f6510 msgr2=0x7fbe700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:10.915 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.913+0000 7fbe7600f700 1 --2- 172.21.15.174:0/3351931018 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe700f6510 0x7fbe700f6930 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fbe60009fb0 tx=0x7fbe60009b30).stop 2023-12-25T21:13:10.916 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.913+0000 7fbe7600f700 1 -- 172.21.15.174:0/3351931018 shutdown_connections 2023-12-25T21:13:10.916 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.913+0000 7fbe7600f700 1 --2- 172.21.15.174:0/3351931018 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe700f6510 0x7fbe700f6930 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:10.916 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.913+0000 7fbe7600f700 1 -- 172.21.15.174:0/3351931018 >> 172.21.15.174:0/3351931018 conn(0x7fbe700f1c70 msgr2=0x7fbe700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:10.916 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.913+0000 7fbe7600f700 1 -- 172.21.15.174:0/3351931018 shutdown_connections 2023-12-25T21:13:10.916 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.913+0000 7fbe7600f700 1 -- 172.21.15.174:0/3351931018 wait complete. 2023-12-25T21:13:10.916 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.913+0000 7fbe7600f700 1 Processor -- start 2023-12-25T21:13:10.916 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.914+0000 7fbe7600f700 1 -- start start 2023-12-25T21:13:10.916 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.914+0000 7fbe7600f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe700f6510 0x7fbe7006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:10.916 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.914+0000 7fbe7600f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe70069e20 con 0x7fbe700f6510 2023-12-25T21:13:10.916 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.914+0000 7fbe6f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe700f6510 0x7fbe7006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:10.916 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.914+0000 7fbe6f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe700f6510 0x7fbe7006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:53634/0 (socket says 172.21.15.174:53634) 2023-12-25T21:13:10.917 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.914+0000 7fbe6f7fe700 1 -- 172.21.15.174:0/3263653688 learned_addr learned my addr 172.21.15.174:0/3263653688 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:10.917 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.914+0000 7fbe6f7fe700 1 -- 172.21.15.174:0/3263653688 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe60005040 con 0x7fbe700f6510 2023-12-25T21:13:10.917 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.915+0000 7fbe6f7fe700 1 --2- 172.21.15.174:0/3263653688 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe700f6510 0x7fbe7006b9a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fbe60013510 tx=0x7fbe6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:10.917 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.915+0000 7fbe57fff700 1 -- 172.21.15.174:0/3263653688 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe60009320 con 0x7fbe700f6510 2023-12-25T21:13:10.917 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.915+0000 7fbe57fff700 1 -- 172.21.15.174:0/3263653688 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe60013f00 con 0x7fbe700f6510 2023-12-25T21:13:10.917 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.915+0000 7fbe7600f700 1 -- 172.21.15.174:0/3263653688 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe70069fc0 con 0x7fbe700f6510 2023-12-25T21:13:10.917 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.915+0000 7fbe57fff700 1 -- 172.21.15.174:0/3263653688 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe60014d90 con 0x7fbe700f6510 2023-12-25T21:13:10.917 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.915+0000 7fbe7600f700 1 -- 172.21.15.174:0/3263653688 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe7006a450 con 0x7fbe700f6510 2023-12-25T21:13:10.918 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.916+0000 7fbe57fff700 1 -- 172.21.15.174:0/3263653688 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbe600095f0 con 0x7fbe700f6510 2023-12-25T21:13:10.918 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.916+0000 7fbe7600f700 1 -- 172.21.15.174:0/3263653688 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe70059100 con 0x7fbe700f6510 2023-12-25T21:13:10.918 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.916+0000 7fbe57fff700 1 --2- 172.21.15.174:0/3263653688 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbe5802ff60 0x7fbe58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:10.918 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.916+0000 7fbe57fff700 1 -- 172.21.15.174:0/3263653688 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbe60041000 con 0x7fbe700f6510 2023-12-25T21:13:10.918 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.916+0000 7fbe6effd700 1 --2- 172.21.15.174:0/3263653688 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbe5802ff60 0x7fbe58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:10.918 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.917+0000 7fbe6effd700 1 --2- 172.21.15.174:0/3263653688 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbe5802ff60 0x7fbe58032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fbe640096a0 tx=0x7fbe64006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:10.922 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:10.921+0000 7fbe57fff700 1 -- 172.21.15.174:0/3263653688 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe6001d7f0 con 0x7fbe700f6510 2023-12-25T21:13:11.127 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:11.127+0000 7fbe7600f700 1 -- 172.21.15.174:0/3263653688 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe700fbf60 con 0x7fbe700f6510 2023-12-25T21:13:11.128 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:11.127+0000 7fbe57fff700 1 -- 172.21.15.174:0/3263653688 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbe60010230 con 0x7fbe700f6510 2023-12-25T21:13:11.128 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:11.128 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:11.130 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:11.129+0000 7fbe7600f700 1 -- 172.21.15.174:0/3263653688 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbe5802ff60 msgr2=0x7fbe58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:11.130 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:11.129+0000 7fbe7600f700 1 --2- 172.21.15.174:0/3263653688 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbe5802ff60 0x7fbe58032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fbe640096a0 tx=0x7fbe64006b40).stop 2023-12-25T21:13:11.130 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:11.129+0000 7fbe7600f700 1 -- 172.21.15.174:0/3263653688 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe700f6510 msgr2=0x7fbe7006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:11.130 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:11.129+0000 7fbe7600f700 1 --2- 172.21.15.174:0/3263653688 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe700f6510 0x7fbe7006b9a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fbe60013510 tx=0x7fbe6000b040).stop 2023-12-25T21:13:11.130 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:11.129+0000 7fbe7600f700 1 -- 172.21.15.174:0/3263653688 shutdown_connections 2023-12-25T21:13:11.130 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:11.129+0000 7fbe7600f700 1 --2- 172.21.15.174:0/3263653688 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbe5802ff60 0x7fbe58032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:11.131 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:11.129+0000 7fbe7600f700 1 --2- 172.21.15.174:0/3263653688 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe700f6510 0x7fbe7006b9a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:11.131 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:11.129+0000 7fbe7600f700 1 -- 172.21.15.174:0/3263653688 >> 172.21.15.174:0/3263653688 conn(0x7fbe700f1c70 msgr2=0x7fbe700f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:11.131 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:11.129+0000 7fbe7600f700 1 -- 172.21.15.174:0/3263653688 shutdown_connections 2023-12-25T21:13:11.131 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:11.129+0000 7fbe7600f700 1 -- 172.21.15.174:0/3263653688 wait complete. 2023-12-25T21:13:11.131 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:11.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:11 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:11.128779+0000 mon.smithi061 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.174:0/3263653688' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:12.553 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:12.553 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:12.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:12 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:10.626226+0000 mgr.smithi061.jxxeke (mgr.14164) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:13.718 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.717+0000 7f7423fff700 1 -- 172.21.15.174:0/3929641539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7424060160 msgr2=0x7f74240f8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:13.718 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.717+0000 7f7423fff700 1 --2- 172.21.15.174:0/3929641539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7424060160 0x7f74240f8fa0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f7414009fb0 tx=0x7f7414009b30).stop 2023-12-25T21:13:13.719 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.717+0000 7f7423fff700 1 -- 172.21.15.174:0/3929641539 shutdown_connections 2023-12-25T21:13:13.719 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.717+0000 7f7423fff700 1 --2- 172.21.15.174:0/3929641539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7424060160 0x7f74240f8fa0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:13.719 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.717+0000 7f7423fff700 1 -- 172.21.15.174:0/3929641539 >> 172.21.15.174:0/3929641539 conn(0x7f74240f09a0 msgr2=0x7f74240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:13.719 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.717+0000 7f7423fff700 1 -- 172.21.15.174:0/3929641539 shutdown_connections 2023-12-25T21:13:13.719 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.717+0000 7f7423fff700 1 -- 172.21.15.174:0/3929641539 wait complete. 2023-12-25T21:13:13.721 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.718+0000 7f7423fff700 1 Processor -- start 2023-12-25T21:13:13.721 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.718+0000 7f7423fff700 1 -- start start 2023-12-25T21:13:13.721 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.718+0000 7f7423fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7424060160 0x7f7424106530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:13.721 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.718+0000 7f7423fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7424106a20 con 0x7f7424060160 2023-12-25T21:13:13.721 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.719+0000 7f7422ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7424060160 0x7f7424106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:13.721 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.719+0000 7f7422ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7424060160 0x7f7424106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:53646/0 (socket says 172.21.15.174:53646) 2023-12-25T21:13:13.721 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.719+0000 7f7422ffd700 1 -- 172.21.15.174:0/3979290204 learned_addr learned my addr 172.21.15.174:0/3979290204 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:13.722 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.719+0000 7f7422ffd700 1 -- 172.21.15.174:0/3979290204 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7414005040 con 0x7f7424060160 2023-12-25T21:13:13.722 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.719+0000 7f7422ffd700 1 --2- 172.21.15.174:0/3979290204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7424060160 0x7f7424106530 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f7414014000 tx=0x7f741400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:13.722 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.719+0000 7f740b7fe700 1 -- 172.21.15.174:0/3979290204 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74140095f0 con 0x7f7424060160 2023-12-25T21:13:13.722 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.719+0000 7f7423fff700 1 -- 172.21.15.174:0/3979290204 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7424106c20 con 0x7f7424060160 2023-12-25T21:13:13.722 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.719+0000 7f740b7fe700 1 -- 172.21.15.174:0/3979290204 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7414007e70 con 0x7f7424060160 2023-12-25T21:13:13.722 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.719+0000 7f740b7fe700 1 -- 172.21.15.174:0/3979290204 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7414014ea0 con 0x7f7424060160 2023-12-25T21:13:13.722 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.719+0000 7f7423fff700 1 -- 172.21.15.174:0/3979290204 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74241070b0 con 0x7f7424060160 2023-12-25T21:13:13.722 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.720+0000 7f740b7fe700 1 -- 172.21.15.174:0/3979290204 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f741401d4b0 con 0x7f7424060160 2023-12-25T21:13:13.722 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.720+0000 7f740b7fe700 1 --2- 172.21.15.174:0/3979290204 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f740c02ffb0 0x7f740c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:13.722 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.720+0000 7f740b7fe700 1 -- 172.21.15.174:0/3979290204 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7414040920 con 0x7f7424060160 2023-12-25T21:13:13.722 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.720+0000 7f7423fff700 1 -- 172.21.15.174:0/3979290204 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7424045ad0 con 0x7f7424060160 2023-12-25T21:13:13.722 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.721+0000 7f74227fc700 1 --2- 172.21.15.174:0/3979290204 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f740c02ffb0 0x7f740c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:13.723 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.721+0000 7f74227fc700 1 --2- 172.21.15.174:0/3979290204 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f740c02ffb0 0x7f740c032470 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f74180096a0 tx=0x7f7418006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:13.726 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.725+0000 7f740b7fe700 1 -- 172.21.15.174:0/3979290204 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f741400f8f0 con 0x7f7424060160 2023-12-25T21:13:13.930 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.929+0000 7f7423fff700 1 -- 172.21.15.174:0/3979290204 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7424107d60 con 0x7f7424060160 2023-12-25T21:13:13.930 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.929+0000 7f740b7fe700 1 -- 172.21.15.174:0/3979290204 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f741402d090 con 0x7f7424060160 2023-12-25T21:13:13.931 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:13.931 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:13.932 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.931+0000 7f7423fff700 1 -- 172.21.15.174:0/3979290204 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f740c02ffb0 msgr2=0x7f740c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:13.932 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.931+0000 7f7423fff700 1 --2- 172.21.15.174:0/3979290204 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f740c02ffb0 0x7f740c032470 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f74180096a0 tx=0x7f7418006b40).stop 2023-12-25T21:13:13.932 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.931+0000 7f7423fff700 1 -- 172.21.15.174:0/3979290204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7424060160 msgr2=0x7f7424106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:13.932 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.931+0000 7f7423fff700 1 --2- 172.21.15.174:0/3979290204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7424060160 0x7f7424106530 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f7414014000 tx=0x7f741400b040).stop 2023-12-25T21:13:13.932 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.931+0000 7f7423fff700 1 -- 172.21.15.174:0/3979290204 shutdown_connections 2023-12-25T21:13:13.932 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.931+0000 7f7423fff700 1 --2- 172.21.15.174:0/3979290204 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f740c02ffb0 0x7f740c032470 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:13.932 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.931+0000 7f7423fff700 1 --2- 172.21.15.174:0/3979290204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7424060160 0x7f7424106530 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:13.932 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.931+0000 7f7423fff700 1 -- 172.21.15.174:0/3979290204 >> 172.21.15.174:0/3979290204 conn(0x7f74240f09a0 msgr2=0x7f74240f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:13.932 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.931+0000 7f7423fff700 1 -- 172.21.15.174:0/3979290204 shutdown_connections 2023-12-25T21:13:13.932 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:13.931+0000 7f7423fff700 1 -- 172.21.15.174:0/3979290204 wait complete. 2023-12-25T21:13:13.933 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:14.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:14 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:12.626631+0000 mgr.smithi061.jxxeke (mgr.14164) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:14.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:14 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:13.931118+0000 mon.smithi061 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.174:0/3979290204' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:15.318 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:15.318 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:16.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.474+0000 7f01b91f5700 1 -- 172.21.15.174:0/3570179202 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01b40f6510 msgr2=0x7f01b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:16.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.474+0000 7f01b91f5700 1 --2- 172.21.15.174:0/3570179202 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01b40f6510 0x7f01b40f6930 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f01a8009fb0 tx=0x7f01a8009b30).stop 2023-12-25T21:13:16.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.474+0000 7f01b91f5700 1 -- 172.21.15.174:0/3570179202 shutdown_connections 2023-12-25T21:13:16.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.474+0000 7f01b91f5700 1 --2- 172.21.15.174:0/3570179202 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01b40f6510 0x7f01b40f6930 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:16.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.474+0000 7f01b91f5700 1 -- 172.21.15.174:0/3570179202 >> 172.21.15.174:0/3570179202 conn(0x7f01b40f1c70 msgr2=0x7f01b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:16.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.475+0000 7f01b91f5700 1 -- 172.21.15.174:0/3570179202 shutdown_connections 2023-12-25T21:13:16.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.475+0000 7f01b91f5700 1 -- 172.21.15.174:0/3570179202 wait complete. 2023-12-25T21:13:16.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.475+0000 7f01b91f5700 1 Processor -- start 2023-12-25T21:13:16.477 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.475+0000 7f01b91f5700 1 -- start start 2023-12-25T21:13:16.477 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.475+0000 7f01b91f5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01b40f6510 0x7f01b4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:16.477 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.475+0000 7f01b91f5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b4106c20 con 0x7f01b40f6510 2023-12-25T21:13:16.477 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.476+0000 7f01b2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01b40f6510 0x7f01b4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:16.477 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.476+0000 7f01b2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01b40f6510 0x7f01b4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:53726/0 (socket says 172.21.15.174:53726) 2023-12-25T21:13:16.477 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.476+0000 7f01b2d9d700 1 -- 172.21.15.174:0/404264839 learned_addr learned my addr 172.21.15.174:0/404264839 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:16.478 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.478+0000 7f01b2d9d700 1 -- 172.21.15.174:0/404264839 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a8005040 con 0x7f01b40f6510 2023-12-25T21:13:16.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.478+0000 7f01b2d9d700 1 --2- 172.21.15.174:0/404264839 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01b40f6510 0x7f01b4106730 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f01a8013510 tx=0x7f01a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:16.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.478+0000 7f019b7fe700 1 -- 172.21.15.174:0/404264839 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01a8009320 con 0x7f01b40f6510 2023-12-25T21:13:16.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.479+0000 7f01b91f5700 1 -- 172.21.15.174:0/404264839 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b4106e20 con 0x7f01b40f6510 2023-12-25T21:13:16.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.479+0000 7f019b7fe700 1 -- 172.21.15.174:0/404264839 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f01a8013f00 con 0x7f01b40f6510 2023-12-25T21:13:16.480 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.479+0000 7f019b7fe700 1 -- 172.21.15.174:0/404264839 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01a8014d90 con 0x7f01b40f6510 2023-12-25T21:13:16.480 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.479+0000 7f01b91f5700 1 -- 172.21.15.174:0/404264839 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b41072b0 con 0x7f01b40f6510 2023-12-25T21:13:16.480 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.479+0000 7f019b7fe700 1 -- 172.21.15.174:0/404264839 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01a80095f0 con 0x7f01b40f6510 2023-12-25T21:13:16.480 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.479+0000 7f01b91f5700 1 -- 172.21.15.174:0/404264839 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b4045ad0 con 0x7f01b40f6510 2023-12-25T21:13:16.480 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.479+0000 7f019b7fe700 1 --2- 172.21.15.174:0/404264839 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f019c02ff60 0x7f019c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:16.480 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.480+0000 7f019b7fe700 1 -- 172.21.15.174:0/404264839 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f01a8040fe0 con 0x7f01b40f6510 2023-12-25T21:13:16.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.480+0000 7f01b259c700 1 --2- 172.21.15.174:0/404264839 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f019c02ff60 0x7f019c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:16.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.480+0000 7f01b259c700 1 --2- 172.21.15.174:0/404264839 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f019c02ff60 0x7f019c032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f01a40096a0 tx=0x7f01a4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:16.485 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.485+0000 7f019b7fe700 1 -- 172.21.15.174:0/404264839 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f01a801d7f0 con 0x7f01b40f6510 2023-12-25T21:13:16.690 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.689+0000 7f01b91f5700 1 -- 172.21.15.174:0/404264839 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01b40fbf60 con 0x7f01b40f6510 2023-12-25T21:13:16.691 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.690+0000 7f019b7fe700 1 -- 172.21.15.174:0/404264839 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f01a8010230 con 0x7f01b40f6510 2023-12-25T21:13:16.691 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:16.691 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:16.692 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.691+0000 7f01b91f5700 1 -- 172.21.15.174:0/404264839 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f019c02ff60 msgr2=0x7f019c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:16.692 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.691+0000 7f01b91f5700 1 --2- 172.21.15.174:0/404264839 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f019c02ff60 0x7f019c032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f01a40096a0 tx=0x7f01a4006b40).stop 2023-12-25T21:13:16.693 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.691+0000 7f01b91f5700 1 -- 172.21.15.174:0/404264839 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01b40f6510 msgr2=0x7f01b4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:16.693 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.691+0000 7f01b91f5700 1 --2- 172.21.15.174:0/404264839 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01b40f6510 0x7f01b4106730 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f01a8013510 tx=0x7f01a800b040).stop 2023-12-25T21:13:16.693 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.691+0000 7f01b91f5700 1 -- 172.21.15.174:0/404264839 shutdown_connections 2023-12-25T21:13:16.693 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.691+0000 7f01b91f5700 1 --2- 172.21.15.174:0/404264839 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f019c02ff60 0x7f019c032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:16.693 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.691+0000 7f01b91f5700 1 --2- 172.21.15.174:0/404264839 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01b40f6510 0x7f01b4106730 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:16.693 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.691+0000 7f01b91f5700 1 -- 172.21.15.174:0/404264839 >> 172.21.15.174:0/404264839 conn(0x7f01b40f1c70 msgr2=0x7f01b40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:16.693 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.691+0000 7f01b91f5700 1 -- 172.21.15.174:0/404264839 shutdown_connections 2023-12-25T21:13:16.693 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:16.691+0000 7f01b91f5700 1 -- 172.21.15.174:0/404264839 wait complete. 2023-12-25T21:13:16.693 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:16.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:16 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:14.627044+0000 mgr.smithi061.jxxeke (mgr.14164) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:17.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:17 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:16.691308+0000 mon.smithi061 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.174:0/404264839' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:18.072 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:18.073 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:18.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:18 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:16.627415+0000 mgr.smithi061.jxxeke (mgr.14164) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:19.250 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.249+0000 7f0b31140700 1 -- 172.21.15.174:0/2128420000 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b2c0f5410 msgr2=0x7f0b2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:19.250 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.249+0000 7f0b31140700 1 --2- 172.21.15.174:0/2128420000 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b2c0f5410 0x7f0b2c0f5830 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f0b14009fb0 tx=0x7f0b14009b30).stop 2023-12-25T21:13:19.250 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.249+0000 7f0b31140700 1 -- 172.21.15.174:0/2128420000 shutdown_connections 2023-12-25T21:13:19.250 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.249+0000 7f0b31140700 1 --2- 172.21.15.174:0/2128420000 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b2c0f5410 0x7f0b2c0f5830 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:19.250 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.249+0000 7f0b31140700 1 -- 172.21.15.174:0/2128420000 >> 172.21.15.174:0/2128420000 conn(0x7f0b2c0f0b70 msgr2=0x7f0b2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:19.250 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.249+0000 7f0b31140700 1 -- 172.21.15.174:0/2128420000 shutdown_connections 2023-12-25T21:13:19.251 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.249+0000 7f0b31140700 1 -- 172.21.15.174:0/2128420000 wait complete. 2023-12-25T21:13:19.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.250+0000 7f0b31140700 1 Processor -- start 2023-12-25T21:13:19.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.250+0000 7f0b31140700 1 -- start start 2023-12-25T21:13:19.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.250+0000 7f0b31140700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b2c0f5410 0x7f0b2c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:19.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.250+0000 7f0b31140700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c060220 con 0x7f0b2c0f5410 2023-12-25T21:13:19.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.250+0000 7f0b2ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b2c0f5410 0x7f0b2c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:19.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.250+0000 7f0b2ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b2c0f5410 0x7f0b2c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:53744/0 (socket says 172.21.15.174:53744) 2023-12-25T21:13:19.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.250+0000 7f0b2ad9d700 1 -- 172.21.15.174:0/1143047118 learned_addr learned my addr 172.21.15.174:0/1143047118 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:19.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.251+0000 7f0b2ad9d700 1 -- 172.21.15.174:0/1143047118 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b14005040 con 0x7f0b2c0f5410 2023-12-25T21:13:19.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.251+0000 7f0b2ad9d700 1 --2- 172.21.15.174:0/1143047118 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b2c0f5410 0x7f0b2c109450 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f0b14013510 tx=0x7f0b1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:19.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.251+0000 7f0b237fe700 1 -- 172.21.15.174:0/1143047118 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b14009320 con 0x7f0b2c0f5410 2023-12-25T21:13:19.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.251+0000 7f0b31140700 1 -- 172.21.15.174:0/1143047118 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2c060420 con 0x7f0b2c0f5410 2023-12-25T21:13:19.254 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.251+0000 7f0b237fe700 1 -- 172.21.15.174:0/1143047118 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b14013f00 con 0x7f0b2c0f5410 2023-12-25T21:13:19.254 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.251+0000 7f0b237fe700 1 -- 172.21.15.174:0/1143047118 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b14014d90 con 0x7f0b2c0f5410 2023-12-25T21:13:19.254 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.251+0000 7f0b31140700 1 -- 172.21.15.174:0/1143047118 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2c109b00 con 0x7f0b2c0f5410 2023-12-25T21:13:19.254 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.252+0000 7f0b237fe700 1 -- 172.21.15.174:0/1143047118 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b140095f0 con 0x7f0b2c0f5410 2023-12-25T21:13:19.254 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.252+0000 7f0b31140700 1 -- 172.21.15.174:0/1143047118 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b2c059100 con 0x7f0b2c0f5410 2023-12-25T21:13:19.254 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.252+0000 7f0b237fe700 1 --2- 172.21.15.174:0/1143047118 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f0b1802ff60 0x7f0b18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:19.254 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.252+0000 7f0b237fe700 1 -- 172.21.15.174:0/1143047118 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b14041000 con 0x7f0b2c0f5410 2023-12-25T21:13:19.254 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.252+0000 7f0b2a59c700 1 --2- 172.21.15.174:0/1143047118 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f0b1802ff60 0x7f0b18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:19.254 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.253+0000 7f0b2a59c700 1 --2- 172.21.15.174:0/1143047118 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f0b1802ff60 0x7f0b18032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f0b1c0096a0 tx=0x7f0b1c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:19.257 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.257+0000 7f0b237fe700 1 -- 172.21.15.174:0/1143047118 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b1401d7f0 con 0x7f0b2c0f5410 2023-12-25T21:13:19.467 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.466+0000 7f0b31140700 1 -- 172.21.15.174:0/1143047118 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b2c045ad0 con 0x7f0b2c0f5410 2023-12-25T21:13:19.468 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.467+0000 7f0b237fe700 1 -- 172.21.15.174:0/1143047118 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0b14010230 con 0x7f0b2c0f5410 2023-12-25T21:13:19.468 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:19.468 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:19.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.469+0000 7f0b31140700 1 -- 172.21.15.174:0/1143047118 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f0b1802ff60 msgr2=0x7f0b18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:19.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.469+0000 7f0b31140700 1 --2- 172.21.15.174:0/1143047118 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f0b1802ff60 0x7f0b18032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f0b1c0096a0 tx=0x7f0b1c006b40).stop 2023-12-25T21:13:19.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.469+0000 7f0b31140700 1 -- 172.21.15.174:0/1143047118 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b2c0f5410 msgr2=0x7f0b2c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:19.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.469+0000 7f0b31140700 1 --2- 172.21.15.174:0/1143047118 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b2c0f5410 0x7f0b2c109450 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f0b14013510 tx=0x7f0b1400b040).stop 2023-12-25T21:13:19.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.469+0000 7f0b31140700 1 -- 172.21.15.174:0/1143047118 shutdown_connections 2023-12-25T21:13:19.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.469+0000 7f0b31140700 1 --2- 172.21.15.174:0/1143047118 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f0b1802ff60 0x7f0b18032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:19.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.469+0000 7f0b31140700 1 --2- 172.21.15.174:0/1143047118 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b2c0f5410 0x7f0b2c109450 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:19.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.469+0000 7f0b31140700 1 -- 172.21.15.174:0/1143047118 >> 172.21.15.174:0/1143047118 conn(0x7f0b2c0f0b70 msgr2=0x7f0b2c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:19.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.469+0000 7f0b31140700 1 -- 172.21.15.174:0/1143047118 shutdown_connections 2023-12-25T21:13:19.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:19.469+0000 7f0b31140700 1 -- 172.21.15.174:0/1143047118 wait complete. 2023-12-25T21:13:19.471 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:20.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:20 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:18.627727+0000 mgr.smithi061.jxxeke (mgr.14164) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:20.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:20 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:19.468614+0000 mon.smithi061 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.174:0/1143047118' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:20.900 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:20.901 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:22.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.007+0000 7f751cd75700 1 -- 172.21.15.174:0/794124413 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180f5410 msgr2=0x7f75180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:22.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.007+0000 7f751cd75700 1 --2- 172.21.15.174:0/794124413 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180f5410 0x7f75180f5830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f7508009960 tx=0x7f7508004d10).stop 2023-12-25T21:13:22.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.007+0000 7f751cd75700 1 -- 172.21.15.174:0/794124413 shutdown_connections 2023-12-25T21:13:22.009 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.007+0000 7f751cd75700 1 --2- 172.21.15.174:0/794124413 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180f5410 0x7f75180f5830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:22.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.007+0000 7f751cd75700 1 -- 172.21.15.174:0/794124413 >> 172.21.15.174:0/794124413 conn(0x7f75180f0b70 msgr2=0x7f75180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:22.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.007+0000 7f751cd75700 1 -- 172.21.15.174:0/794124413 shutdown_connections 2023-12-25T21:13:22.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.007+0000 7f751cd75700 1 -- 172.21.15.174:0/794124413 wait complete. 2023-12-25T21:13:22.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.008+0000 7f751cd75700 1 Processor -- start 2023-12-25T21:13:22.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.008+0000 7f751cd75700 1 -- start start 2023-12-25T21:13:22.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.008+0000 7f751cd75700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180f5410 0x7f7518109440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:22.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.008+0000 7f751cd75700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7518060240 con 0x7f75180f5410 2023-12-25T21:13:22.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.008+0000 7f751659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180f5410 0x7f7518109440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:22.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.009+0000 7f751659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180f5410 0x7f7518109440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:53770/0 (socket says 172.21.15.174:53770) 2023-12-25T21:13:22.010 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.009+0000 7f751659c700 1 -- 172.21.15.174:0/316450483 learned_addr learned my addr 172.21.15.174:0/316450483 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:22.011 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.009+0000 7f751659c700 1 -- 172.21.15.174:0/316450483 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7508009640 con 0x7f75180f5410 2023-12-25T21:13:22.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.009+0000 7f751659c700 1 --2- 172.21.15.174:0/316450483 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180f5410 0x7f7518109440 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f7508004a10 tx=0x7f7508004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:22.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.009+0000 7f750effd700 1 -- 172.21.15.174:0/316450483 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7508018070 con 0x7f75180f5410 2023-12-25T21:13:22.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.009+0000 7f751cd75700 1 -- 172.21.15.174:0/316450483 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7518060440 con 0x7f75180f5410 2023-12-25T21:13:22.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.009+0000 7f750effd700 1 -- 172.21.15.174:0/316450483 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7508007e70 con 0x7f75180f5410 2023-12-25T21:13:22.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.009+0000 7f750effd700 1 -- 172.21.15.174:0/316450483 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f750801d420 con 0x7f75180f5410 2023-12-25T21:13:22.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.009+0000 7f751cd75700 1 -- 172.21.15.174:0/316450483 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7518109af0 con 0x7f75180f5410 2023-12-25T21:13:22.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.010+0000 7f750effd700 1 -- 172.21.15.174:0/316450483 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f750801d580 con 0x7f75180f5410 2023-12-25T21:13:22.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.010+0000 7f751cd75700 1 -- 172.21.15.174:0/316450483 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7518045ad0 con 0x7f75180f5410 2023-12-25T21:13:22.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.010+0000 7f750effd700 1 --2- 172.21.15.174:0/316450483 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f750402ff60 0x7f7504032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:22.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.010+0000 7f750effd700 1 -- 172.21.15.174:0/316450483 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f75080415e0 con 0x7f75180f5410 2023-12-25T21:13:22.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.010+0000 7f7515d9b700 1 --2- 172.21.15.174:0/316450483 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f750402ff60 0x7f7504032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:22.012 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.011+0000 7f7515d9b700 1 --2- 172.21.15.174:0/316450483 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f750402ff60 0x7f7504032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f75000096a0 tx=0x7f7500006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:22.015 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.015+0000 7f750effd700 1 -- 172.21.15.174:0/316450483 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f750801d930 con 0x7f75180f5410 2023-12-25T21:13:22.224 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.223+0000 7f751cd75700 1 -- 172.21.15.174:0/316450483 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f751810a320 con 0x7f75180f5410 2023-12-25T21:13:22.224 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.224+0000 7f750effd700 1 -- 172.21.15.174:0/316450483 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f750801bb60 con 0x7f75180f5410 2023-12-25T21:13:22.224 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:22.224 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:22.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.225+0000 7f751cd75700 1 -- 172.21.15.174:0/316450483 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f750402ff60 msgr2=0x7f7504032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:22.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.225+0000 7f751cd75700 1 --2- 172.21.15.174:0/316450483 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f750402ff60 0x7f7504032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f75000096a0 tx=0x7f7500006b40).stop 2023-12-25T21:13:22.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.225+0000 7f751cd75700 1 -- 172.21.15.174:0/316450483 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180f5410 msgr2=0x7f7518109440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:22.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.225+0000 7f751cd75700 1 --2- 172.21.15.174:0/316450483 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180f5410 0x7f7518109440 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f7508004a10 tx=0x7f7508004d10).stop 2023-12-25T21:13:22.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.225+0000 7f751cd75700 1 -- 172.21.15.174:0/316450483 shutdown_connections 2023-12-25T21:13:22.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.225+0000 7f751cd75700 1 --2- 172.21.15.174:0/316450483 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f750402ff60 0x7f7504032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:22.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.225+0000 7f751cd75700 1 --2- 172.21.15.174:0/316450483 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180f5410 0x7f7518109440 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:22.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.225+0000 7f751cd75700 1 -- 172.21.15.174:0/316450483 >> 172.21.15.174:0/316450483 conn(0x7f75180f0b70 msgr2=0x7f75180f1680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:22.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.225+0000 7f751cd75700 1 -- 172.21.15.174:0/316450483 shutdown_connections 2023-12-25T21:13:22.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:22.225+0000 7f751cd75700 1 -- 172.21.15.174:0/316450483 wait complete. 2023-12-25T21:13:22.227 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:22.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:22 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:20.628085+0000 mgr.smithi061.jxxeke (mgr.14164) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:22.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:22 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:22.225173+0000 mon.smithi061 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.174:0/316450483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:23.627 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:23.627 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:24.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:24 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:22.628379+0000 mgr.smithi061.jxxeke (mgr.14164) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:24.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.810+0000 7f7153fff700 1 -- 172.21.15.174:0/2203231445 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71540f51a0 msgr2=0x7f71540f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:24.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.810+0000 7f7153fff700 1 --2- 172.21.15.174:0/2203231445 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71540f51a0 0x7f71540f55c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f7144009fb0 tx=0x7f7144009b30).stop 2023-12-25T21:13:24.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.810+0000 7f7153fff700 1 -- 172.21.15.174:0/2203231445 shutdown_connections 2023-12-25T21:13:24.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.810+0000 7f7153fff700 1 --2- 172.21.15.174:0/2203231445 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71540f51a0 0x7f71540f55c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:24.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.810+0000 7f7153fff700 1 -- 172.21.15.174:0/2203231445 >> 172.21.15.174:0/2203231445 conn(0x7f71540f09a0 msgr2=0x7f71540f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:24.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.810+0000 7f7153fff700 1 -- 172.21.15.174:0/2203231445 shutdown_connections 2023-12-25T21:13:24.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.810+0000 7f7153fff700 1 -- 172.21.15.174:0/2203231445 wait complete. 2023-12-25T21:13:24.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.811+0000 7f7153fff700 1 Processor -- start 2023-12-25T21:13:24.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.811+0000 7f7153fff700 1 -- start start 2023-12-25T21:13:24.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.811+0000 7f7153fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71540f51a0 0x7f71541091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:24.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.811+0000 7f7153fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7154060240 con 0x7f71540f51a0 2023-12-25T21:13:24.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.811+0000 7f7152ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71540f51a0 0x7f71541091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:24.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.812+0000 7f7152ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71540f51a0 0x7f71541091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:53782/0 (socket says 172.21.15.174:53782) 2023-12-25T21:13:24.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.812+0000 7f7152ffd700 1 -- 172.21.15.174:0/3393488833 learned_addr learned my addr 172.21.15.174:0/3393488833 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:24.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.812+0000 7f7152ffd700 1 -- 172.21.15.174:0/3393488833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7144005040 con 0x7f71540f51a0 2023-12-25T21:13:24.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.812+0000 7f7152ffd700 1 --2- 172.21.15.174:0/3393488833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71540f51a0 0x7f71541091b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f7144013510 tx=0x7f714400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:24.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.812+0000 7f713b7fe700 1 -- 172.21.15.174:0/3393488833 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7144009320 con 0x7f71540f51a0 2023-12-25T21:13:24.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.812+0000 7f7153fff700 1 -- 172.21.15.174:0/3393488833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7154060440 con 0x7f71540f51a0 2023-12-25T21:13:24.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.812+0000 7f7153fff700 1 -- 172.21.15.174:0/3393488833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7154109860 con 0x7f71540f51a0 2023-12-25T21:13:24.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.812+0000 7f713b7fe700 1 -- 172.21.15.174:0/3393488833 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7144013f00 con 0x7f71540f51a0 2023-12-25T21:13:24.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.812+0000 7f713b7fe700 1 -- 172.21.15.174:0/3393488833 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7144014cc0 con 0x7f71540f51a0 2023-12-25T21:13:24.815 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.813+0000 7f7153fff700 1 -- 172.21.15.174:0/3393488833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7134004fa0 con 0x7f71540f51a0 2023-12-25T21:13:24.815 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.813+0000 7f713b7fe700 1 -- 172.21.15.174:0/3393488833 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7144014420 con 0x7f71540f51a0 2023-12-25T21:13:24.815 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.813+0000 7f713b7fe700 1 --2- 172.21.15.174:0/3393488833 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f713c02ff60 0x7f713c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:24.816 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.813+0000 7f713b7fe700 1 -- 172.21.15.174:0/3393488833 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7144041160 con 0x7f71540f51a0 2023-12-25T21:13:24.816 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.814+0000 7f71527fc700 1 --2- 172.21.15.174:0/3393488833 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f713c02ff60 0x7f713c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:24.816 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.814+0000 7f71527fc700 1 --2- 172.21.15.174:0/3393488833 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f713c02ff60 0x7f713c032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f71480096a0 tx=0x7f7148006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:24.819 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:24.818+0000 7f713b7fe700 1 -- 172.21.15.174:0/3393488833 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f71440095f0 con 0x7f71540f51a0 2023-12-25T21:13:25.023 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:25.022+0000 7f7153fff700 1 -- 172.21.15.174:0/3393488833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7134005670 con 0x7f71540f51a0 2023-12-25T21:13:25.024 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:25.023+0000 7f713b7fe700 1 -- 172.21.15.174:0/3393488833 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f71440095f0 con 0x7f71540f51a0 2023-12-25T21:13:25.024 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:25.024 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:25.025 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:25.024+0000 7f7153fff700 1 -- 172.21.15.174:0/3393488833 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f713c02ff60 msgr2=0x7f713c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:25.026 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:25.025+0000 7f7153fff700 1 --2- 172.21.15.174:0/3393488833 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f713c02ff60 0x7f713c032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f71480096a0 tx=0x7f7148006b40).stop 2023-12-25T21:13:25.026 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:25.025+0000 7f7153fff700 1 -- 172.21.15.174:0/3393488833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71540f51a0 msgr2=0x7f71541091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:25.026 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:25.025+0000 7f7153fff700 1 --2- 172.21.15.174:0/3393488833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71540f51a0 0x7f71541091b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f7144013510 tx=0x7f714400b040).stop 2023-12-25T21:13:25.026 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:25.025+0000 7f7153fff700 1 -- 172.21.15.174:0/3393488833 shutdown_connections 2023-12-25T21:13:25.026 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:25.025+0000 7f7153fff700 1 --2- 172.21.15.174:0/3393488833 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f713c02ff60 0x7f713c032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:25.026 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:25.025+0000 7f7153fff700 1 --2- 172.21.15.174:0/3393488833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71540f51a0 0x7f71541091b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:25.026 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:25.025+0000 7f7153fff700 1 -- 172.21.15.174:0/3393488833 >> 172.21.15.174:0/3393488833 conn(0x7f71540f09a0 msgr2=0x7f71540f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:25.026 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:25.025+0000 7f7153fff700 1 -- 172.21.15.174:0/3393488833 shutdown_connections 2023-12-25T21:13:25.026 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:25.025+0000 7f7153fff700 1 -- 172.21.15.174:0/3393488833 wait complete. 2023-12-25T21:13:25.026 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:25.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:25 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:25.024510+0000 mon.smithi061 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.174:0/3393488833' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:26.384 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:26.384 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:26.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:26 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:24.628693+0000 mgr.smithi061.jxxeke (mgr.14164) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:27.645 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.642+0000 7f1358c14700 1 -- 172.21.15.174:0/2982123508 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13540f62a0 msgr2=0x7f13540f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:27.645 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.642+0000 7f1358c14700 1 --2- 172.21.15.174:0/2982123508 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13540f62a0 0x7f13540f66c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f1344009fb0 tx=0x7f1344009b30).stop 2023-12-25T21:13:27.645 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.642+0000 7f1358c14700 1 -- 172.21.15.174:0/2982123508 shutdown_connections 2023-12-25T21:13:27.645 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.642+0000 7f1358c14700 1 --2- 172.21.15.174:0/2982123508 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13540f62a0 0x7f13540f66c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:27.645 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.642+0000 7f1358c14700 1 -- 172.21.15.174:0/2982123508 >> 172.21.15.174:0/2982123508 conn(0x7f13540f1aa0 msgr2=0x7f13540f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:27.645 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.642+0000 7f1358c14700 1 -- 172.21.15.174:0/2982123508 shutdown_connections 2023-12-25T21:13:27.646 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.642+0000 7f1358c14700 1 -- 172.21.15.174:0/2982123508 wait complete. 2023-12-25T21:13:27.646 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.642+0000 7f1358c14700 1 Processor -- start 2023-12-25T21:13:27.646 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.643+0000 7f1358c14700 1 -- start start 2023-12-25T21:13:27.646 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.643+0000 7f1358c14700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13540f62a0 0x7f13541064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:27.646 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.643+0000 7f1358c14700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13541069a0 con 0x7f13540f62a0 2023-12-25T21:13:27.646 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.643+0000 7f13537fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13540f62a0 0x7f13541064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:27.646 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.643+0000 7f13537fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13540f62a0 0x7f13541064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:42060/0 (socket says 172.21.15.174:42060) 2023-12-25T21:13:27.646 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.643+0000 7f13537fe700 1 -- 172.21.15.174:0/3241085919 learned_addr learned my addr 172.21.15.174:0/3241085919 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:27.646 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.643+0000 7f13537fe700 1 -- 172.21.15.174:0/3241085919 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1344005040 con 0x7f13540f62a0 2023-12-25T21:13:27.646 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.644+0000 7f13537fe700 1 --2- 172.21.15.174:0/3241085919 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13540f62a0 0x7f13541064b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f1344013510 tx=0x7f134400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:27.646 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.644+0000 7f133bfff700 1 -- 172.21.15.174:0/3241085919 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1344009320 con 0x7f13540f62a0 2023-12-25T21:13:27.646 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.644+0000 7f1358c14700 1 -- 172.21.15.174:0/3241085919 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1354106ba0 con 0x7f13540f62a0 2023-12-25T21:13:27.647 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.644+0000 7f133bfff700 1 -- 172.21.15.174:0/3241085919 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1344013f00 con 0x7f13540f62a0 2023-12-25T21:13:27.647 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.644+0000 7f133bfff700 1 -- 172.21.15.174:0/3241085919 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1344014dc0 con 0x7f13540f62a0 2023-12-25T21:13:27.647 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.644+0000 7f1358c14700 1 -- 172.21.15.174:0/3241085919 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1354107030 con 0x7f13540f62a0 2023-12-25T21:13:27.648 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.645+0000 7f133bfff700 1 -- 172.21.15.174:0/3241085919 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13440095f0 con 0x7f13540f62a0 2023-12-25T21:13:27.648 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.645+0000 7f1358c14700 1 -- 172.21.15.174:0/3241085919 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1354045ad0 con 0x7f13540f62a0 2023-12-25T21:13:27.648 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.645+0000 7f133bfff700 1 --2- 172.21.15.174:0/3241085919 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f133c02ff60 0x7f133c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:27.648 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.645+0000 7f133bfff700 1 -- 172.21.15.174:0/3241085919 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1344041000 con 0x7f13540f62a0 2023-12-25T21:13:27.648 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.645+0000 7f1352ffd700 1 --2- 172.21.15.174:0/3241085919 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f133c02ff60 0x7f133c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:27.648 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.646+0000 7f1352ffd700 1 --2- 172.21.15.174:0/3241085919 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f133c02ff60 0x7f133c032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f13480096a0 tx=0x7f1348006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:27.650 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.650+0000 7f133bfff700 1 -- 172.21.15.174:0/3241085919 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1344014420 con 0x7f13540f62a0 2023-12-25T21:13:27.855 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.854+0000 7f1358c14700 1 -- 172.21.15.174:0/3241085919 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1354107ee0 con 0x7f13540f62a0 2023-12-25T21:13:27.856 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.855+0000 7f133bfff700 1 -- 172.21.15.174:0/3241085919 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1344010230 con 0x7f13540f62a0 2023-12-25T21:13:27.856 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:27.856 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:27.858 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.857+0000 7f1358c14700 1 -- 172.21.15.174:0/3241085919 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f133c02ff60 msgr2=0x7f133c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:27.858 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.857+0000 7f1358c14700 1 --2- 172.21.15.174:0/3241085919 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f133c02ff60 0x7f133c032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f13480096a0 tx=0x7f1348006b40).stop 2023-12-25T21:13:27.858 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.857+0000 7f1358c14700 1 -- 172.21.15.174:0/3241085919 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13540f62a0 msgr2=0x7f13541064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:27.858 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.857+0000 7f1358c14700 1 --2- 172.21.15.174:0/3241085919 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13540f62a0 0x7f13541064b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f1344013510 tx=0x7f134400b040).stop 2023-12-25T21:13:27.858 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.857+0000 7f1358c14700 1 -- 172.21.15.174:0/3241085919 shutdown_connections 2023-12-25T21:13:27.858 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.857+0000 7f1358c14700 1 --2- 172.21.15.174:0/3241085919 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f133c02ff60 0x7f133c032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:27.858 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.857+0000 7f1358c14700 1 --2- 172.21.15.174:0/3241085919 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13540f62a0 0x7f13541064b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:27.858 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.857+0000 7f1358c14700 1 -- 172.21.15.174:0/3241085919 >> 172.21.15.174:0/3241085919 conn(0x7f13540f1aa0 msgr2=0x7f13540f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:27.858 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.857+0000 7f1358c14700 1 -- 172.21.15.174:0/3241085919 shutdown_connections 2023-12-25T21:13:27.858 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:27.857+0000 7f1358c14700 1 -- 172.21.15.174:0/3241085919 wait complete. 2023-12-25T21:13:27.859 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:28.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:28 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:26.629021+0000 mgr.smithi061.jxxeke (mgr.14164) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:28.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:28 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:27.856472+0000 mon.smithi061 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.174:0/3241085919' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:29.255 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:29.256 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:30.445 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.444+0000 7fbc3ae2a700 1 -- 172.21.15.174:0/584284837 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc340f78e0 msgr2=0x7fbc340f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:30.448 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.444+0000 7fbc3ae2a700 1 --2- 172.21.15.174:0/584284837 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc340f78e0 0x7fbc340f7d00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fbc30009fb0 tx=0x7fbc30009b30).stop 2023-12-25T21:13:30.448 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.444+0000 7fbc3ae2a700 1 -- 172.21.15.174:0/584284837 shutdown_connections 2023-12-25T21:13:30.448 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.444+0000 7fbc3ae2a700 1 --2- 172.21.15.174:0/584284837 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc340f78e0 0x7fbc340f7d00 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:30.448 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.444+0000 7fbc3ae2a700 1 -- 172.21.15.174:0/584284837 >> 172.21.15.174:0/584284837 conn(0x7fbc340f0120 msgr2=0x7fbc340f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:30.449 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.444+0000 7fbc3ae2a700 1 -- 172.21.15.174:0/584284837 shutdown_connections 2023-12-25T21:13:30.449 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.444+0000 7fbc3ae2a700 1 -- 172.21.15.174:0/584284837 wait complete. 2023-12-25T21:13:30.449 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.444+0000 7fbc3ae2a700 1 Processor -- start 2023-12-25T21:13:30.449 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.444+0000 7fbc3ae2a700 1 -- start start 2023-12-25T21:13:30.449 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.444+0000 7fbc3ae2a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc340f78e0 0x7fbc341064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:30.449 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.444+0000 7fbc3ae2a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc341069d0 con 0x7fbc340f78e0 2023-12-25T21:13:30.450 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.445+0000 7fbc39e28700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc340f78e0 0x7fbc341064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:30.450 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.445+0000 7fbc39e28700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc340f78e0 0x7fbc341064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:42080/0 (socket says 172.21.15.174:42080) 2023-12-25T21:13:30.451 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.445+0000 7fbc39e28700 1 -- 172.21.15.174:0/3633946681 learned_addr learned my addr 172.21.15.174:0/3633946681 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:30.451 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.445+0000 7fbc39e28700 1 -- 172.21.15.174:0/3633946681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc30005040 con 0x7fbc340f78e0 2023-12-25T21:13:30.451 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.445+0000 7fbc39e28700 1 --2- 172.21.15.174:0/3633946681 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc340f78e0 0x7fbc341064e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fbc30013510 tx=0x7fbc3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:30.451 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.445+0000 7fbc2a7fc700 1 -- 172.21.15.174:0/3633946681 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc30009320 con 0x7fbc340f78e0 2023-12-25T21:13:30.451 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.445+0000 7fbc3ae2a700 1 -- 172.21.15.174:0/3633946681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc34106bd0 con 0x7fbc340f78e0 2023-12-25T21:13:30.451 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.445+0000 7fbc2a7fc700 1 -- 172.21.15.174:0/3633946681 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc30013f00 con 0x7fbc340f78e0 2023-12-25T21:13:30.451 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.445+0000 7fbc2a7fc700 1 -- 172.21.15.174:0/3633946681 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc30014dc0 con 0x7fbc340f78e0 2023-12-25T21:13:30.451 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.445+0000 7fbc3ae2a700 1 -- 172.21.15.174:0/3633946681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc34107060 con 0x7fbc340f78e0 2023-12-25T21:13:30.451 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.446+0000 7fbc3ae2a700 1 -- 172.21.15.174:0/3633946681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc34045ad0 con 0x7fbc340f78e0 2023-12-25T21:13:30.451 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.446+0000 7fbc2a7fc700 1 -- 172.21.15.174:0/3633946681 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc300095f0 con 0x7fbc340f78e0 2023-12-25T21:13:30.451 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.446+0000 7fbc2a7fc700 1 --2- 172.21.15.174:0/3633946681 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbc2002ff60 0x7fbc20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:30.451 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.447+0000 7fbc2a7fc700 1 -- 172.21.15.174:0/3633946681 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc300410e0 con 0x7fbc340f78e0 2023-12-25T21:13:30.451 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.447+0000 7fbc39627700 1 --2- 172.21.15.174:0/3633946681 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbc2002ff60 0x7fbc20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:30.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.447+0000 7fbc39627700 1 --2- 172.21.15.174:0/3633946681 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbc2002ff60 0x7fbc20032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fbc240096a0 tx=0x7fbc24006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:30.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.451+0000 7fbc2a7fc700 1 -- 172.21.15.174:0/3633946681 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc30014420 con 0x7fbc340f78e0 2023-12-25T21:13:30.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.658+0000 7fbc3ae2a700 1 -- 172.21.15.174:0/3633946681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc34107eb0 con 0x7fbc340f78e0 2023-12-25T21:13:30.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.658+0000 7fbc2a7fc700 1 -- 172.21.15.174:0/3633946681 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbc3001c920 con 0x7fbc340f78e0 2023-12-25T21:13:30.659 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:30.659 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:30.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.660+0000 7fbc3ae2a700 1 -- 172.21.15.174:0/3633946681 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbc2002ff60 msgr2=0x7fbc20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:30.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.660+0000 7fbc3ae2a700 1 --2- 172.21.15.174:0/3633946681 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbc2002ff60 0x7fbc20032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fbc240096a0 tx=0x7fbc24006b40).stop 2023-12-25T21:13:30.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.660+0000 7fbc3ae2a700 1 -- 172.21.15.174:0/3633946681 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc340f78e0 msgr2=0x7fbc341064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:30.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.660+0000 7fbc3ae2a700 1 --2- 172.21.15.174:0/3633946681 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc340f78e0 0x7fbc341064e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fbc30013510 tx=0x7fbc3000b040).stop 2023-12-25T21:13:30.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.660+0000 7fbc3ae2a700 1 -- 172.21.15.174:0/3633946681 shutdown_connections 2023-12-25T21:13:30.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.660+0000 7fbc3ae2a700 1 --2- 172.21.15.174:0/3633946681 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbc2002ff60 0x7fbc20032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:30.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.660+0000 7fbc3ae2a700 1 --2- 172.21.15.174:0/3633946681 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc340f78e0 0x7fbc341064e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:30.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.660+0000 7fbc3ae2a700 1 -- 172.21.15.174:0/3633946681 >> 172.21.15.174:0/3633946681 conn(0x7fbc340f0120 msgr2=0x7fbc340f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:30.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.660+0000 7fbc3ae2a700 1 -- 172.21.15.174:0/3633946681 shutdown_connections 2023-12-25T21:13:30.662 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:30.660+0000 7fbc3ae2a700 1 -- 172.21.15.174:0/3633946681 wait complete. 2023-12-25T21:13:30.662 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:30.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:30 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:28.629372+0000 mgr.smithi061.jxxeke (mgr.14164) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:31.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:31 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:30.659989+0000 mon.smithi061 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.174:0/3633946681' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:32.034 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:32.034 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:32.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:32 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:30.629767+0000 mgr.smithi061.jxxeke (mgr.14164) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:33.271 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.269+0000 7f0fcdb7f700 1 -- 172.21.15.174:0/2073979147 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0fc80601e0 msgr2=0x7f0fc80f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:33.271 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.269+0000 7f0fcdb7f700 1 --2- 172.21.15.174:0/2073979147 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0fc80601e0 0x7f0fc80f9190 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f0fbc009fb0 tx=0x7f0fbc009b30).stop 2023-12-25T21:13:33.271 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.269+0000 7f0fcdb7f700 1 -- 172.21.15.174:0/2073979147 shutdown_connections 2023-12-25T21:13:33.271 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.269+0000 7f0fcdb7f700 1 --2- 172.21.15.174:0/2073979147 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0fc80601e0 0x7f0fc80f9190 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:33.271 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.269+0000 7f0fcdb7f700 1 -- 172.21.15.174:0/2073979147 >> 172.21.15.174:0/2073979147 conn(0x7f0fc80f0b70 msgr2=0x7f0fc80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:33.271 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.269+0000 7f0fcdb7f700 1 -- 172.21.15.174:0/2073979147 shutdown_connections 2023-12-25T21:13:33.271 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.269+0000 7f0fcdb7f700 1 -- 172.21.15.174:0/2073979147 wait complete. 2023-12-25T21:13:33.271 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.270+0000 7f0fcdb7f700 1 Processor -- start 2023-12-25T21:13:33.271 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.270+0000 7f0fcdb7f700 1 -- start start 2023-12-25T21:13:33.271 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.270+0000 7f0fcdb7f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0fc80601e0 0x7f0fc8108900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:33.271 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.270+0000 7f0fcdb7f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc8108df0 con 0x7f0fc80601e0 2023-12-25T21:13:33.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.271+0000 7f0fc77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0fc80601e0 0x7f0fc8108900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:33.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.271+0000 7f0fc77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0fc80601e0 0x7f0fc8108900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:42094/0 (socket says 172.21.15.174:42094) 2023-12-25T21:13:33.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.271+0000 7f0fc77fe700 1 -- 172.21.15.174:0/4012071866 learned_addr learned my addr 172.21.15.174:0/4012071866 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:33.273 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.272+0000 7f0fc77fe700 1 -- 172.21.15.174:0/4012071866 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fbc005040 con 0x7f0fc80601e0 2023-12-25T21:13:33.273 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.272+0000 7f0fc77fe700 1 --2- 172.21.15.174:0/4012071866 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0fc80601e0 0x7f0fc8108900 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f0fbc004dd0 tx=0x7f0fbc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:33.273 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.273+0000 7f0faffff700 1 -- 172.21.15.174:0/4012071866 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fbc018070 con 0x7f0fc80601e0 2023-12-25T21:13:33.274 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.273+0000 7f0fcdb7f700 1 -- 172.21.15.174:0/4012071866 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc8108ff0 con 0x7f0fc80601e0 2023-12-25T21:13:33.274 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.273+0000 7f0faffff700 1 -- 172.21.15.174:0/4012071866 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0fbc013410 con 0x7f0fc80601e0 2023-12-25T21:13:33.274 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.273+0000 7f0faffff700 1 -- 172.21.15.174:0/4012071866 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fbc01c3c0 con 0x7f0fc80601e0 2023-12-25T21:13:33.274 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.273+0000 7f0fcdb7f700 1 -- 172.21.15.174:0/4012071866 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc8109480 con 0x7f0fc80601e0 2023-12-25T21:13:33.274 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.273+0000 7f0faffff700 1 -- 172.21.15.174:0/4012071866 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0fbc01d420 con 0x7f0fc80601e0 2023-12-25T21:13:33.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.274+0000 7f0fcdb7f700 1 -- 172.21.15.174:0/4012071866 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc8045ad0 con 0x7f0fc80601e0 2023-12-25T21:13:33.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.274+0000 7f0faffff700 1 --2- 172.21.15.174:0/4012071866 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f0fb002ff60 0x7f0fb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:33.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.274+0000 7f0faffff700 1 -- 172.21.15.174:0/4012071866 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0fbc0413d0 con 0x7f0fc80601e0 2023-12-25T21:13:33.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.274+0000 7f0fc6ffd700 1 --2- 172.21.15.174:0/4012071866 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f0fb002ff60 0x7f0fb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:33.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.274+0000 7f0fc6ffd700 1 --2- 172.21.15.174:0/4012071866 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f0fb002ff60 0x7f0fb0032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f0fb80096a0 tx=0x7f0fb8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:33.280 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.279+0000 7f0faffff700 1 -- 172.21.15.174:0/4012071866 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0fbc01c630 con 0x7f0fc80601e0 2023-12-25T21:13:33.483 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.482+0000 7f0fcdb7f700 1 -- 172.21.15.174:0/4012071866 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0fc810a2d0 con 0x7f0fc80601e0 2023-12-25T21:13:33.484 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.483+0000 7f0faffff700 1 -- 172.21.15.174:0/4012071866 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0fbc01b370 con 0x7f0fc80601e0 2023-12-25T21:13:33.484 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:33.485 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:33.486 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.485+0000 7f0fcdb7f700 1 -- 172.21.15.174:0/4012071866 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f0fb002ff60 msgr2=0x7f0fb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:33.486 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.485+0000 7f0fcdb7f700 1 --2- 172.21.15.174:0/4012071866 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f0fb002ff60 0x7f0fb0032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f0fb80096a0 tx=0x7f0fb8006b40).stop 2023-12-25T21:13:33.486 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.485+0000 7f0fcdb7f700 1 -- 172.21.15.174:0/4012071866 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0fc80601e0 msgr2=0x7f0fc8108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:33.486 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.485+0000 7f0fcdb7f700 1 --2- 172.21.15.174:0/4012071866 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0fc80601e0 0x7f0fc8108900 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f0fbc004dd0 tx=0x7f0fbc009b30).stop 2023-12-25T21:13:33.487 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.485+0000 7f0fcdb7f700 1 -- 172.21.15.174:0/4012071866 shutdown_connections 2023-12-25T21:13:33.487 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.485+0000 7f0fcdb7f700 1 --2- 172.21.15.174:0/4012071866 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f0fb002ff60 0x7f0fb0032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:33.487 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.485+0000 7f0fcdb7f700 1 --2- 172.21.15.174:0/4012071866 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0fc80601e0 0x7f0fc8108900 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:33.487 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.485+0000 7f0fcdb7f700 1 -- 172.21.15.174:0/4012071866 >> 172.21.15.174:0/4012071866 conn(0x7f0fc80f0b70 msgr2=0x7f0fc80f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:33.487 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.485+0000 7f0fcdb7f700 1 -- 172.21.15.174:0/4012071866 shutdown_connections 2023-12-25T21:13:33.487 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:33.485+0000 7f0fcdb7f700 1 -- 172.21.15.174:0/4012071866 wait complete. 2023-12-25T21:13:33.487 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:34.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:34 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:32.630145+0000 mgr.smithi061.jxxeke (mgr.14164) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:34.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:34 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:33.484794+0000 mon.smithi061 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.174:0/4012071866' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:34.868 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:34.869 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:36.028 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.027+0000 7f3d15932700 1 -- 172.21.15.174:0/2720092681 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d100f5410 msgr2=0x7f3d100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:36.028 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.027+0000 7f3d15932700 1 --2- 172.21.15.174:0/2720092681 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d100f5410 0x7f3d100f5830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f3d04009fb0 tx=0x7f3d04009b30).stop 2023-12-25T21:13:36.028 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.027+0000 7f3d15932700 1 -- 172.21.15.174:0/2720092681 shutdown_connections 2023-12-25T21:13:36.029 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.027+0000 7f3d15932700 1 --2- 172.21.15.174:0/2720092681 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d100f5410 0x7f3d100f5830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:36.029 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.027+0000 7f3d15932700 1 -- 172.21.15.174:0/2720092681 >> 172.21.15.174:0/2720092681 conn(0x7f3d100f0b70 msgr2=0x7f3d100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:36.029 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.027+0000 7f3d15932700 1 -- 172.21.15.174:0/2720092681 shutdown_connections 2023-12-25T21:13:36.029 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.027+0000 7f3d15932700 1 -- 172.21.15.174:0/2720092681 wait complete. 2023-12-25T21:13:36.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.028+0000 7f3d15932700 1 Processor -- start 2023-12-25T21:13:36.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.028+0000 7f3d15932700 1 -- start start 2023-12-25T21:13:36.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.028+0000 7f3d15932700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d100f5410 0x7f3d10105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:36.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.028+0000 7f3d15932700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d10105b20 con 0x7f3d100f5410 2023-12-25T21:13:36.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.028+0000 7f3d0effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d100f5410 0x7f3d10105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:36.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.028+0000 7f3d0effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d100f5410 0x7f3d10105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:42118/0 (socket says 172.21.15.174:42118) 2023-12-25T21:13:36.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.028+0000 7f3d0effd700 1 -- 172.21.15.174:0/2396905485 learned_addr learned my addr 172.21.15.174:0/2396905485 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:36.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.029+0000 7f3d0effd700 1 -- 172.21.15.174:0/2396905485 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d04005040 con 0x7f3d100f5410 2023-12-25T21:13:36.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.029+0000 7f3d0effd700 1 --2- 172.21.15.174:0/2396905485 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d100f5410 0x7f3d10105630 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f3d04013510 tx=0x7f3d0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:36.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.029+0000 7f3cf77fe700 1 -- 172.21.15.174:0/2396905485 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d04009320 con 0x7f3d100f5410 2023-12-25T21:13:36.031 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.029+0000 7f3d15932700 1 -- 172.21.15.174:0/2396905485 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d10105d20 con 0x7f3d100f5410 2023-12-25T21:13:36.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.029+0000 7f3cf77fe700 1 -- 172.21.15.174:0/2396905485 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d04013f00 con 0x7f3d100f5410 2023-12-25T21:13:36.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.029+0000 7f3cf77fe700 1 -- 172.21.15.174:0/2396905485 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d04014d90 con 0x7f3d100f5410 2023-12-25T21:13:36.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.029+0000 7f3d15932700 1 -- 172.21.15.174:0/2396905485 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d101061b0 con 0x7f3d100f5410 2023-12-25T21:13:36.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.030+0000 7f3cf77fe700 1 -- 172.21.15.174:0/2396905485 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d040095f0 con 0x7f3d100f5410 2023-12-25T21:13:36.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.030+0000 7f3d15932700 1 -- 172.21.15.174:0/2396905485 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d10045ad0 con 0x7f3d100f5410 2023-12-25T21:13:36.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.030+0000 7f3cf77fe700 1 --2- 172.21.15.174:0/2396905485 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3cf802ff60 0x7f3cf8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:36.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.030+0000 7f3cf77fe700 1 -- 172.21.15.174:0/2396905485 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d04042dc0 con 0x7f3d100f5410 2023-12-25T21:13:36.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.030+0000 7f3d0e7fc700 1 --2- 172.21.15.174:0/2396905485 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3cf802ff60 0x7f3cf8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:36.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.031+0000 7f3d0e7fc700 1 --2- 172.21.15.174:0/2396905485 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3cf802ff60 0x7f3cf8032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f3d000096a0 tx=0x7f3d00006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:36.035 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.035+0000 7f3cf77fe700 1 -- 172.21.15.174:0/2396905485 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d04014420 con 0x7f3d100f5410 2023-12-25T21:13:36.241 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.240+0000 7f3d15932700 1 -- 172.21.15.174:0/2396905485 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d100fae60 con 0x7f3d100f5410 2023-12-25T21:13:36.241 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.241+0000 7f3cf77fe700 1 -- 172.21.15.174:0/2396905485 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3d04011290 con 0x7f3d100f5410 2023-12-25T21:13:36.242 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:36.242 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:36.243 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.242+0000 7f3d15932700 1 -- 172.21.15.174:0/2396905485 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3cf802ff60 msgr2=0x7f3cf8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:36.243 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.242+0000 7f3d15932700 1 --2- 172.21.15.174:0/2396905485 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3cf802ff60 0x7f3cf8032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f3d000096a0 tx=0x7f3d00006b40).stop 2023-12-25T21:13:36.243 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.242+0000 7f3d15932700 1 -- 172.21.15.174:0/2396905485 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d100f5410 msgr2=0x7f3d10105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:36.243 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.242+0000 7f3d15932700 1 --2- 172.21.15.174:0/2396905485 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d100f5410 0x7f3d10105630 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f3d04013510 tx=0x7f3d0400b040).stop 2023-12-25T21:13:36.244 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.242+0000 7f3d15932700 1 -- 172.21.15.174:0/2396905485 shutdown_connections 2023-12-25T21:13:36.244 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.242+0000 7f3d15932700 1 --2- 172.21.15.174:0/2396905485 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3cf802ff60 0x7f3cf8032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:36.244 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.242+0000 7f3d15932700 1 --2- 172.21.15.174:0/2396905485 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d100f5410 0x7f3d10105630 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:36.244 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.242+0000 7f3d15932700 1 -- 172.21.15.174:0/2396905485 >> 172.21.15.174:0/2396905485 conn(0x7f3d100f0b70 msgr2=0x7f3d100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:36.244 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.242+0000 7f3d15932700 1 -- 172.21.15.174:0/2396905485 shutdown_connections 2023-12-25T21:13:36.244 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:36.242+0000 7f3d15932700 1 -- 172.21.15.174:0/2396905485 wait complete. 2023-12-25T21:13:36.244 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:36.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:36 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:34.630514+0000 mgr.smithi061.jxxeke (mgr.14164) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:36.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:36 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:36.242332+0000 mon.smithi061 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.174:0/2396905485' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:37.646 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:37.647 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:38.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:38 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:36.630811+0000 mgr.smithi061.jxxeke (mgr.14164) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:38.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.834+0000 7f97095e6700 1 -- 172.21.15.174:0/704800360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97040f5410 msgr2=0x7f97040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:38.840 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.834+0000 7f97095e6700 1 --2- 172.21.15.174:0/704800360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97040f5410 0x7f97040f5830 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f96f8009fb0 tx=0x7f96f8009b30).stop 2023-12-25T21:13:38.841 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.834+0000 7f97095e6700 1 -- 172.21.15.174:0/704800360 shutdown_connections 2023-12-25T21:13:38.841 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.834+0000 7f97095e6700 1 --2- 172.21.15.174:0/704800360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97040f5410 0x7f97040f5830 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:38.841 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.834+0000 7f97095e6700 1 -- 172.21.15.174:0/704800360 >> 172.21.15.174:0/704800360 conn(0x7f97040f0b70 msgr2=0x7f97040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:38.841 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.834+0000 7f97095e6700 1 -- 172.21.15.174:0/704800360 shutdown_connections 2023-12-25T21:13:38.841 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.834+0000 7f97095e6700 1 -- 172.21.15.174:0/704800360 wait complete. 2023-12-25T21:13:38.842 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.835+0000 7f97095e6700 1 Processor -- start 2023-12-25T21:13:38.842 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.835+0000 7f97095e6700 1 -- start start 2023-12-25T21:13:38.842 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.835+0000 7f97095e6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97040f5410 0x7f9704109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:38.842 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.835+0000 7f97095e6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9704060220 con 0x7f97040f5410 2023-12-25T21:13:38.842 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.835+0000 7f9702ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97040f5410 0x7f9704109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:38.842 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.835+0000 7f9702ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97040f5410 0x7f9704109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:46072/0 (socket says 172.21.15.174:46072) 2023-12-25T21:13:38.843 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.835+0000 7f9702ffd700 1 -- 172.21.15.174:0/2792861573 learned_addr learned my addr 172.21.15.174:0/2792861573 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:38.843 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.836+0000 7f9702ffd700 1 -- 172.21.15.174:0/2792861573 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f8005040 con 0x7f97040f5410 2023-12-25T21:13:38.843 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.836+0000 7f9702ffd700 1 --2- 172.21.15.174:0/2792861573 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97040f5410 0x7f9704109420 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f96f8013510 tx=0x7f96f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:38.843 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.836+0000 7f96eb7fe700 1 -- 172.21.15.174:0/2792861573 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96f8009320 con 0x7f97040f5410 2023-12-25T21:13:38.843 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.836+0000 7f97095e6700 1 -- 172.21.15.174:0/2792861573 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9704060420 con 0x7f97040f5410 2023-12-25T21:13:38.843 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.836+0000 7f96eb7fe700 1 -- 172.21.15.174:0/2792861573 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f96f8013f00 con 0x7f97040f5410 2023-12-25T21:13:38.843 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.836+0000 7f96eb7fe700 1 -- 172.21.15.174:0/2792861573 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96f8014d90 con 0x7f97040f5410 2023-12-25T21:13:38.844 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.836+0000 7f97095e6700 1 -- 172.21.15.174:0/2792861573 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9704109ad0 con 0x7f97040f5410 2023-12-25T21:13:38.844 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.837+0000 7f96eb7fe700 1 -- 172.21.15.174:0/2792861573 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f96f8014420 con 0x7f97040f5410 2023-12-25T21:13:38.844 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.837+0000 7f97095e6700 1 -- 172.21.15.174:0/2792861573 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9704059100 con 0x7f97040f5410 2023-12-25T21:13:38.844 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.837+0000 7f96eb7fe700 1 --2- 172.21.15.174:0/2792861573 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f96ec02ffb0 0x7f96ec032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:38.844 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.837+0000 7f96eb7fe700 1 -- 172.21.15.174:0/2792861573 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f96f8040ff0 con 0x7f97040f5410 2023-12-25T21:13:38.845 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.837+0000 7f97027fc700 1 --2- 172.21.15.174:0/2792861573 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f96ec02ffb0 0x7f96ec032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:38.845 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.838+0000 7f97027fc700 1 --2- 172.21.15.174:0/2792861573 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f96ec02ffb0 0x7f96ec032470 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f96f40096a0 tx=0x7f96f4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:38.845 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:38.842+0000 7f96eb7fe700 1 -- 172.21.15.174:0/2792861573 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f96f806d2b0 con 0x7f97040f5410 2023-12-25T21:13:39.051 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:39.050+0000 7f97095e6700 1 -- 172.21.15.174:0/2792861573 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9704045ad0 con 0x7f97040f5410 2023-12-25T21:13:39.051 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:39.050+0000 7f96eb7fe700 1 -- 172.21.15.174:0/2792861573 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f96f80095f0 con 0x7f97040f5410 2023-12-25T21:13:39.051 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:39.052 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:39.053 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:39.052+0000 7f97095e6700 1 -- 172.21.15.174:0/2792861573 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f96ec02ffb0 msgr2=0x7f96ec032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:39.053 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:39.052+0000 7f97095e6700 1 --2- 172.21.15.174:0/2792861573 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f96ec02ffb0 0x7f96ec032470 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f96f40096a0 tx=0x7f96f4006b40).stop 2023-12-25T21:13:39.053 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:39.052+0000 7f97095e6700 1 -- 172.21.15.174:0/2792861573 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97040f5410 msgr2=0x7f9704109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:39.053 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:39.052+0000 7f97095e6700 1 --2- 172.21.15.174:0/2792861573 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97040f5410 0x7f9704109420 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f96f8013510 tx=0x7f96f800b040).stop 2023-12-25T21:13:39.053 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:39.052+0000 7f97095e6700 1 -- 172.21.15.174:0/2792861573 shutdown_connections 2023-12-25T21:13:39.053 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:39.052+0000 7f97095e6700 1 --2- 172.21.15.174:0/2792861573 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f96ec02ffb0 0x7f96ec032470 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:39.054 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:39.052+0000 7f97095e6700 1 --2- 172.21.15.174:0/2792861573 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97040f5410 0x7f9704109420 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:39.054 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:39.052+0000 7f97095e6700 1 -- 172.21.15.174:0/2792861573 >> 172.21.15.174:0/2792861573 conn(0x7f97040f0b70 msgr2=0x7f97040f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:39.054 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:39.052+0000 7f97095e6700 1 -- 172.21.15.174:0/2792861573 shutdown_connections 2023-12-25T21:13:39.054 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:39.052+0000 7f97095e6700 1 -- 172.21.15.174:0/2792861573 wait complete. 2023-12-25T21:13:39.054 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:39.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:39 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:39.052040+0000 mon.smithi061 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.174:0/2792861573' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:40.484 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:40.484 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:40.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:40 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:38.631031+0000 mgr.smithi061.jxxeke (mgr.14164) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:41.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.654+0000 7fe095231700 1 -- 172.21.15.174:0/971033726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900f6510 msgr2=0x7fe0900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:41.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.654+0000 7fe095231700 1 --2- 172.21.15.174:0/971033726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900f6510 0x7fe0900f6930 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fe084009fb0 tx=0x7fe084009b30).stop 2023-12-25T21:13:41.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.654+0000 7fe095231700 1 -- 172.21.15.174:0/971033726 shutdown_connections 2023-12-25T21:13:41.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.654+0000 7fe095231700 1 --2- 172.21.15.174:0/971033726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900f6510 0x7fe0900f6930 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:41.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.654+0000 7fe095231700 1 -- 172.21.15.174:0/971033726 >> 172.21.15.174:0/971033726 conn(0x7fe0900f1c70 msgr2=0x7fe0900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:41.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.654+0000 7fe095231700 1 -- 172.21.15.174:0/971033726 shutdown_connections 2023-12-25T21:13:41.655 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.654+0000 7fe095231700 1 -- 172.21.15.174:0/971033726 wait complete. 2023-12-25T21:13:41.657 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.655+0000 7fe095231700 1 Processor -- start 2023-12-25T21:13:41.657 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.655+0000 7fe095231700 1 -- start start 2023-12-25T21:13:41.657 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.655+0000 7fe095231700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900f6510 0x7fe090106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:41.657 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.655+0000 7fe095231700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe090106c20 con 0x7fe0900f6510 2023-12-25T21:13:41.657 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.655+0000 7fe08ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900f6510 0x7fe090106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:41.657 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.655+0000 7fe08ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900f6510 0x7fe090106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:46086/0 (socket says 172.21.15.174:46086) 2023-12-25T21:13:41.657 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.655+0000 7fe08ed9d700 1 -- 172.21.15.174:0/4000422060 learned_addr learned my addr 172.21.15.174:0/4000422060 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:41.657 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.656+0000 7fe08ed9d700 1 -- 172.21.15.174:0/4000422060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe084005040 con 0x7fe0900f6510 2023-12-25T21:13:41.657 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.656+0000 7fe08ed9d700 1 --2- 172.21.15.174:0/4000422060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900f6510 0x7fe090106730 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fe084013510 tx=0x7fe08400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:41.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.656+0000 7fe0777fe700 1 -- 172.21.15.174:0/4000422060 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe084009320 con 0x7fe0900f6510 2023-12-25T21:13:41.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.656+0000 7fe0777fe700 1 -- 172.21.15.174:0/4000422060 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe084013f00 con 0x7fe0900f6510 2023-12-25T21:13:41.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.656+0000 7fe0777fe700 1 -- 172.21.15.174:0/4000422060 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe084014d90 con 0x7fe0900f6510 2023-12-25T21:13:41.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.656+0000 7fe095231700 1 -- 172.21.15.174:0/4000422060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe090106e20 con 0x7fe0900f6510 2023-12-25T21:13:41.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.656+0000 7fe095231700 1 -- 172.21.15.174:0/4000422060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0901072b0 con 0x7fe0900f6510 2023-12-25T21:13:41.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.657+0000 7fe0777fe700 1 -- 172.21.15.174:0/4000422060 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0840095f0 con 0x7fe0900f6510 2023-12-25T21:13:41.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.657+0000 7fe095231700 1 -- 172.21.15.174:0/4000422060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe090045ad0 con 0x7fe0900f6510 2023-12-25T21:13:41.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.657+0000 7fe0777fe700 1 --2- 172.21.15.174:0/4000422060 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe07802ff60 0x7fe078032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:41.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.657+0000 7fe0777fe700 1 -- 172.21.15.174:0/4000422060 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe084041080 con 0x7fe0900f6510 2023-12-25T21:13:41.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.657+0000 7fe08e59c700 1 --2- 172.21.15.174:0/4000422060 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe07802ff60 0x7fe078032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:41.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.658+0000 7fe08e59c700 1 --2- 172.21.15.174:0/4000422060 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe07802ff60 0x7fe078032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fe0800096a0 tx=0x7fe080006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:41.662 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.662+0000 7fe0777fe700 1 -- 172.21.15.174:0/4000422060 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe08401d7f0 con 0x7fe0900f6510 2023-12-25T21:13:41.869 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.869+0000 7fe095231700 1 -- 172.21.15.174:0/4000422060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0900fbf60 con 0x7fe0900f6510 2023-12-25T21:13:41.870 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.869+0000 7fe0777fe700 1 -- 172.21.15.174:0/4000422060 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe084010230 con 0x7fe0900f6510 2023-12-25T21:13:41.870 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:41.870 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:41.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.871+0000 7fe095231700 1 -- 172.21.15.174:0/4000422060 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe07802ff60 msgr2=0x7fe078032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:41.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.871+0000 7fe095231700 1 --2- 172.21.15.174:0/4000422060 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe07802ff60 0x7fe078032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fe0800096a0 tx=0x7fe080006b40).stop 2023-12-25T21:13:41.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.871+0000 7fe095231700 1 -- 172.21.15.174:0/4000422060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900f6510 msgr2=0x7fe090106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:41.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.871+0000 7fe095231700 1 --2- 172.21.15.174:0/4000422060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900f6510 0x7fe090106730 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fe084013510 tx=0x7fe08400b040).stop 2023-12-25T21:13:41.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.871+0000 7fe095231700 1 -- 172.21.15.174:0/4000422060 shutdown_connections 2023-12-25T21:13:41.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.871+0000 7fe095231700 1 --2- 172.21.15.174:0/4000422060 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe07802ff60 0x7fe078032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:41.872 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.871+0000 7fe095231700 1 --2- 172.21.15.174:0/4000422060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900f6510 0x7fe090106730 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:41.873 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.871+0000 7fe095231700 1 -- 172.21.15.174:0/4000422060 >> 172.21.15.174:0/4000422060 conn(0x7fe0900f1c70 msgr2=0x7fe0900f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:41.873 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.871+0000 7fe095231700 1 -- 172.21.15.174:0/4000422060 shutdown_connections 2023-12-25T21:13:41.873 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:41.871+0000 7fe095231700 1 -- 172.21.15.174:0/4000422060 wait complete. 2023-12-25T21:13:41.873 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:42.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:42 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:40.631323+0000 mgr.smithi061.jxxeke (mgr.14164) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:42.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:42 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:41.870985+0000 mon.smithi061 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.174:0/4000422060' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:43.255 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:43.256 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:44.478 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.476+0000 7f8f6cd4d700 1 -- 172.21.15.174:0/2183977226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f680f6510 msgr2=0x7f8f680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:44.478 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.476+0000 7f8f6cd4d700 1 --2- 172.21.15.174:0/2183977226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f680f6510 0x7f8f680f6930 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f8f50009fb0 tx=0x7f8f50009b30).stop 2023-12-25T21:13:44.478 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.477+0000 7f8f6cd4d700 1 -- 172.21.15.174:0/2183977226 shutdown_connections 2023-12-25T21:13:44.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.477+0000 7f8f6cd4d700 1 --2- 172.21.15.174:0/2183977226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f680f6510 0x7f8f680f6930 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:44.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.477+0000 7f8f6cd4d700 1 -- 172.21.15.174:0/2183977226 >> 172.21.15.174:0/2183977226 conn(0x7f8f680f1c70 msgr2=0x7f8f680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:44.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.477+0000 7f8f6cd4d700 1 -- 172.21.15.174:0/2183977226 shutdown_connections 2023-12-25T21:13:44.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.477+0000 7f8f6cd4d700 1 -- 172.21.15.174:0/2183977226 wait complete. 2023-12-25T21:13:44.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.477+0000 7f8f6cd4d700 1 Processor -- start 2023-12-25T21:13:44.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.477+0000 7f8f6cd4d700 1 -- start start 2023-12-25T21:13:44.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.477+0000 7f8f6cd4d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f680f6510 0x7f8f68106770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:44.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.477+0000 7f8f6cd4d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f68106c60 con 0x7f8f680f6510 2023-12-25T21:13:44.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.477+0000 7f8f6659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f680f6510 0x7f8f68106770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:44.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.478+0000 7f8f6659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f680f6510 0x7f8f68106770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:46104/0 (socket says 172.21.15.174:46104) 2023-12-25T21:13:44.479 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.478+0000 7f8f6659c700 1 -- 172.21.15.174:0/395706217 learned_addr learned my addr 172.21.15.174:0/395706217 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:44.480 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.478+0000 7f8f6659c700 1 -- 172.21.15.174:0/395706217 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f50005040 con 0x7f8f680f6510 2023-12-25T21:13:44.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.478+0000 7f8f6659c700 1 --2- 172.21.15.174:0/395706217 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f680f6510 0x7f8f68106770 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f8f50000c00 tx=0x7f8f5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:44.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.478+0000 7f8f5effd700 1 -- 172.21.15.174:0/395706217 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f500095f0 con 0x7f8f680f6510 2023-12-25T21:13:44.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.478+0000 7f8f6cd4d700 1 -- 172.21.15.174:0/395706217 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f68106e60 con 0x7f8f680f6510 2023-12-25T21:13:44.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.478+0000 7f8f5effd700 1 -- 172.21.15.174:0/395706217 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f50007e70 con 0x7f8f680f6510 2023-12-25T21:13:44.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.478+0000 7f8f5effd700 1 -- 172.21.15.174:0/395706217 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f50014da0 con 0x7f8f680f6510 2023-12-25T21:13:44.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.478+0000 7f8f6cd4d700 1 -- 172.21.15.174:0/395706217 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f681072f0 con 0x7f8f680f6510 2023-12-25T21:13:44.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.479+0000 7f8f5effd700 1 -- 172.21.15.174:0/395706217 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f50025070 con 0x7f8f680f6510 2023-12-25T21:13:44.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.479+0000 7f8f6cd4d700 1 -- 172.21.15.174:0/395706217 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f68045ad0 con 0x7f8f680f6510 2023-12-25T21:13:44.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.479+0000 7f8f5effd700 1 --2- 172.21.15.174:0/395706217 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8f5402ffb0 0x7f8f54032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:44.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.479+0000 7f8f5effd700 1 -- 172.21.15.174:0/395706217 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f500409c0 con 0x7f8f680f6510 2023-12-25T21:13:44.481 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.480+0000 7f8f65d9b700 1 --2- 172.21.15.174:0/395706217 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8f5402ffb0 0x7f8f54032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:44.482 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.481+0000 7f8f65d9b700 1 --2- 172.21.15.174:0/395706217 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8f5402ffb0 0x7f8f54032470 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f8f580096a0 tx=0x7f8f58006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:44.484 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.484+0000 7f8f5effd700 1 -- 172.21.15.174:0/395706217 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f5000f8d0 con 0x7f8f680f6510 2023-12-25T21:13:44.689 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.688+0000 7f8f6cd4d700 1 -- 172.21.15.174:0/395706217 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f681081a0 con 0x7f8f680f6510 2023-12-25T21:13:44.689 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.690+0000 7f8f5effd700 1 -- 172.21.15.174:0/395706217 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8f5000f300 con 0x7f8f680f6510 2023-12-25T21:13:44.690 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:44.690 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:44.691 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.690+0000 7f8f6cd4d700 1 -- 172.21.15.174:0/395706217 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8f5402ffb0 msgr2=0x7f8f54032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:44.692 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.690+0000 7f8f6cd4d700 1 --2- 172.21.15.174:0/395706217 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8f5402ffb0 0x7f8f54032470 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f8f580096a0 tx=0x7f8f58006b40).stop 2023-12-25T21:13:44.692 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.690+0000 7f8f6cd4d700 1 -- 172.21.15.174:0/395706217 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f680f6510 msgr2=0x7f8f68106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:44.692 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.690+0000 7f8f6cd4d700 1 --2- 172.21.15.174:0/395706217 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f680f6510 0x7f8f68106770 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f8f50000c00 tx=0x7f8f5000b040).stop 2023-12-25T21:13:44.692 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.690+0000 7f8f6cd4d700 1 -- 172.21.15.174:0/395706217 shutdown_connections 2023-12-25T21:13:44.692 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.690+0000 7f8f6cd4d700 1 --2- 172.21.15.174:0/395706217 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8f5402ffb0 0x7f8f54032470 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:44.692 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.690+0000 7f8f6cd4d700 1 --2- 172.21.15.174:0/395706217 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f680f6510 0x7f8f68106770 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:44.692 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.690+0000 7f8f6cd4d700 1 -- 172.21.15.174:0/395706217 >> 172.21.15.174:0/395706217 conn(0x7f8f680f1c70 msgr2=0x7f8f680f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:44.692 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.690+0000 7f8f6cd4d700 1 -- 172.21.15.174:0/395706217 shutdown_connections 2023-12-25T21:13:44.692 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:44.690+0000 7f8f6cd4d700 1 -- 172.21.15.174:0/395706217 wait complete. 2023-12-25T21:13:44.692 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:44.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:44 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:42.631696+0000 mgr.smithi061.jxxeke (mgr.14164) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:45.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:45 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:44.690253+0000 mon.smithi061 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.174:0/395706217' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:46.067 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:46.067 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:46.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:46 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:44.631977+0000 mgr.smithi061.jxxeke (mgr.14164) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:47.252 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.252+0000 7f0b4d089700 1 -- 172.21.15.174:0/1678105650 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b48060280 msgr2=0x7f0b480f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:47.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.252+0000 7f0b4d089700 1 --2- 172.21.15.174:0/1678105650 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b48060280 0x7f0b480f90f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f0b3c009fb0 tx=0x7f0b3c009b30).stop 2023-12-25T21:13:47.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.252+0000 7f0b4d089700 1 -- 172.21.15.174:0/1678105650 shutdown_connections 2023-12-25T21:13:47.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.252+0000 7f0b4d089700 1 --2- 172.21.15.174:0/1678105650 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b48060280 0x7f0b480f90f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:47.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.252+0000 7f0b4d089700 1 -- 172.21.15.174:0/1678105650 >> 172.21.15.174:0/1678105650 conn(0x7f0b480f0b70 msgr2=0x7f0b480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:47.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.252+0000 7f0b4d089700 1 -- 172.21.15.174:0/1678105650 shutdown_connections 2023-12-25T21:13:47.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.252+0000 7f0b4d089700 1 -- 172.21.15.174:0/1678105650 wait complete. 2023-12-25T21:13:47.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.253+0000 7f0b4d089700 1 Processor -- start 2023-12-25T21:13:47.253 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.253+0000 7f0b4d089700 1 -- start start 2023-12-25T21:13:47.254 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.253+0000 7f0b4d089700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b48060280 0x7f0b48106690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:47.254 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.253+0000 7f0b4d089700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b48106b80 con 0x7f0b48060280 2023-12-25T21:13:47.254 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.253+0000 7f0b46d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b48060280 0x7f0b48106690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:47.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.253+0000 7f0b46d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b48060280 0x7f0b48106690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:50860/0 (socket says 172.21.15.174:50860) 2023-12-25T21:13:47.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.253+0000 7f0b46d9d700 1 -- 172.21.15.174:0/72793517 learned_addr learned my addr 172.21.15.174:0/72793517 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:47.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.254+0000 7f0b46d9d700 1 -- 172.21.15.174:0/72793517 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3c005040 con 0x7f0b48060280 2023-12-25T21:13:47.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.254+0000 7f0b46d9d700 1 --2- 172.21.15.174:0/72793517 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b48060280 0x7f0b48106690 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f0b3c013510 tx=0x7f0b3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:47.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.254+0000 7f0b2f7fe700 1 -- 172.21.15.174:0/72793517 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b3c009320 con 0x7f0b48060280 2023-12-25T21:13:47.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.254+0000 7f0b4d089700 1 -- 172.21.15.174:0/72793517 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b48106d80 con 0x7f0b48060280 2023-12-25T21:13:47.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.254+0000 7f0b2f7fe700 1 -- 172.21.15.174:0/72793517 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b3c013f00 con 0x7f0b48060280 2023-12-25T21:13:47.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.254+0000 7f0b2f7fe700 1 -- 172.21.15.174:0/72793517 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b3c014dc0 con 0x7f0b48060280 2023-12-25T21:13:47.255 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.254+0000 7f0b4d089700 1 -- 172.21.15.174:0/72793517 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b48107210 con 0x7f0b48060280 2023-12-25T21:13:47.256 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.255+0000 7f0b2f7fe700 1 -- 172.21.15.174:0/72793517 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b3c014420 con 0x7f0b48060280 2023-12-25T21:13:47.256 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.255+0000 7f0b4d089700 1 -- 172.21.15.174:0/72793517 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b48045ad0 con 0x7f0b48060280 2023-12-25T21:13:47.256 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.255+0000 7f0b2f7fe700 1 --2- 172.21.15.174:0/72793517 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f0b3002ff60 0x7f0b30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:47.256 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.255+0000 7f0b2f7fe700 1 -- 172.21.15.174:0/72793517 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b3c041100 con 0x7f0b48060280 2023-12-25T21:13:47.256 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.256+0000 7f0b4659c700 1 --2- 172.21.15.174:0/72793517 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f0b3002ff60 0x7f0b30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:47.256 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.257+0000 7f0b4659c700 1 --2- 172.21.15.174:0/72793517 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f0b3002ff60 0x7f0b30032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f0b380096a0 tx=0x7f0b38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:47.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.260+0000 7f0b2f7fe700 1 -- 172.21.15.174:0/72793517 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b3c0095f0 con 0x7f0b48060280 2023-12-25T21:13:47.467 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.467+0000 7f0b4d089700 1 -- 172.21.15.174:0/72793517 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b48108060 con 0x7f0b48060280 2023-12-25T21:13:47.467 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.467+0000 7f0b2f7fe700 1 -- 172.21.15.174:0/72793517 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0b3c01cb10 con 0x7f0b48060280 2023-12-25T21:13:47.467 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:47.467 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:47.469 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.469+0000 7f0b4d089700 1 -- 172.21.15.174:0/72793517 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f0b3002ff60 msgr2=0x7f0b30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:47.469 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.469+0000 7f0b4d089700 1 --2- 172.21.15.174:0/72793517 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f0b3002ff60 0x7f0b30032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f0b380096a0 tx=0x7f0b38006b40).stop 2023-12-25T21:13:47.469 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.469+0000 7f0b4d089700 1 -- 172.21.15.174:0/72793517 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b48060280 msgr2=0x7f0b48106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:47.469 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.469+0000 7f0b4d089700 1 --2- 172.21.15.174:0/72793517 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b48060280 0x7f0b48106690 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f0b3c013510 tx=0x7f0b3c00b040).stop 2023-12-25T21:13:47.469 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.469+0000 7f0b4d089700 1 -- 172.21.15.174:0/72793517 shutdown_connections 2023-12-25T21:13:47.469 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.469+0000 7f0b4d089700 1 --2- 172.21.15.174:0/72793517 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f0b3002ff60 0x7f0b30032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:47.469 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.469+0000 7f0b4d089700 1 --2- 172.21.15.174:0/72793517 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b48060280 0x7f0b48106690 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:47.469 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.469+0000 7f0b4d089700 1 -- 172.21.15.174:0/72793517 >> 172.21.15.174:0/72793517 conn(0x7f0b480f0b70 msgr2=0x7f0b480f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:47.469 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.469+0000 7f0b4d089700 1 -- 172.21.15.174:0/72793517 shutdown_connections 2023-12-25T21:13:47.469 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:47.469+0000 7f0b4d089700 1 -- 172.21.15.174:0/72793517 wait complete. 2023-12-25T21:13:47.470 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:47.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:47 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:46.632278+0000 mgr.smithi061.jxxeke (mgr.14164) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:48.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:48 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:47.467911+0000 mon.smithi061 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.174:0/72793517' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:48.847 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:48.847 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:49.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:49 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:48.632667+0000 mgr.smithi061.jxxeke (mgr.14164) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:50.026 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.025+0000 7f92ab621700 1 -- 172.21.15.174:0/3108108504 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92a40f4800 msgr2=0x7f92a40f4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:50.026 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.025+0000 7f92ab621700 1 --2- 172.21.15.174:0/3108108504 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92a40f4800 0x7f92a40f4c20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f92a0009fb0 tx=0x7f92a0009b30).stop 2023-12-25T21:13:50.026 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.025+0000 7f92ab621700 1 -- 172.21.15.174:0/3108108504 shutdown_connections 2023-12-25T21:13:50.026 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.025+0000 7f92ab621700 1 --2- 172.21.15.174:0/3108108504 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92a40f4800 0x7f92a40f4c20 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:50.026 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.025+0000 7f92ab621700 1 -- 172.21.15.174:0/3108108504 >> 172.21.15.174:0/3108108504 conn(0x7f92a40f0260 msgr2=0x7f92a40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:50.026 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.025+0000 7f92ab621700 1 -- 172.21.15.174:0/3108108504 shutdown_connections 2023-12-25T21:13:50.026 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.025+0000 7f92ab621700 1 -- 172.21.15.174:0/3108108504 wait complete. 2023-12-25T21:13:50.026 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.025+0000 7f92ab621700 1 Processor -- start 2023-12-25T21:13:50.027 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.026+0000 7f92ab621700 1 -- start start 2023-12-25T21:13:50.027 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.026+0000 7f92ab621700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92a40f4800 0x7f92a41066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:50.027 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.026+0000 7f92ab621700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a4106ba0 con 0x7f92a40f4800 2023-12-25T21:13:50.027 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.026+0000 7f92a93bd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92a40f4800 0x7f92a41066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:50.027 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.026+0000 7f92a93bd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92a40f4800 0x7f92a41066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:50868/0 (socket says 172.21.15.174:50868) 2023-12-25T21:13:50.027 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.026+0000 7f92a93bd700 1 -- 172.21.15.174:0/3554189366 learned_addr learned my addr 172.21.15.174:0/3554189366 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:50.028 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.027+0000 7f92a93bd700 1 -- 172.21.15.174:0/3554189366 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a0005040 con 0x7f92a40f4800 2023-12-25T21:13:50.028 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.027+0000 7f92a93bd700 1 --2- 172.21.15.174:0/3554189366 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92a40f4800 0x7f92a41066b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f92a0013510 tx=0x7f92a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:50.028 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.027+0000 7f9299ffb700 1 -- 172.21.15.174:0/3554189366 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92a0009320 con 0x7f92a40f4800 2023-12-25T21:13:50.028 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.027+0000 7f92ab621700 1 -- 172.21.15.174:0/3554189366 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92a4106da0 con 0x7f92a40f4800 2023-12-25T21:13:50.029 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.027+0000 7f9299ffb700 1 -- 172.21.15.174:0/3554189366 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f92a0013f00 con 0x7f92a40f4800 2023-12-25T21:13:50.029 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.027+0000 7f9299ffb700 1 -- 172.21.15.174:0/3554189366 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92a0014dc0 con 0x7f92a40f4800 2023-12-25T21:13:50.029 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.027+0000 7f92ab621700 1 -- 172.21.15.174:0/3554189366 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92a4107230 con 0x7f92a40f4800 2023-12-25T21:13:50.029 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.028+0000 7f9299ffb700 1 -- 172.21.15.174:0/3554189366 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f92a001d3d0 con 0x7f92a40f4800 2023-12-25T21:13:50.029 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.028+0000 7f9299ffb700 1 --2- 172.21.15.174:0/3554189366 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f929002ff60 0x7f9290032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:50.029 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.028+0000 7f9299ffb700 1 -- 172.21.15.174:0/3554189366 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f92a00413b0 con 0x7f92a40f4800 2023-12-25T21:13:50.029 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.028+0000 7f92ab621700 1 -- 172.21.15.174:0/3554189366 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a4045ad0 con 0x7f92a40f4800 2023-12-25T21:13:50.029 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.028+0000 7f92a8bbc700 1 --2- 172.21.15.174:0/3554189366 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f929002ff60 0x7f9290032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:50.029 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.029+0000 7f92a8bbc700 1 --2- 172.21.15.174:0/3554189366 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f929002ff60 0x7f9290032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f92940096a0 tx=0x7f9294006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:50.033 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.033+0000 7f9299ffb700 1 -- 172.21.15.174:0/3554189366 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f92a001c930 con 0x7f92a40f4800 2023-12-25T21:13:50.242 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.243+0000 7f92ab621700 1 -- 172.21.15.174:0/3554189366 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92a41080e0 con 0x7f92a40f4800 2023-12-25T21:13:50.243 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.243+0000 7f9299ffb700 1 -- 172.21.15.174:0/3554189366 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f92a002b030 con 0x7f92a40f4800 2023-12-25T21:13:50.243 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:50.243 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:50.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.245+0000 7f92ab621700 1 -- 172.21.15.174:0/3554189366 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f929002ff60 msgr2=0x7f9290032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:50.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.245+0000 7f92ab621700 1 --2- 172.21.15.174:0/3554189366 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f929002ff60 0x7f9290032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f92940096a0 tx=0x7f9294006b40).stop 2023-12-25T21:13:50.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.245+0000 7f92ab621700 1 -- 172.21.15.174:0/3554189366 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92a40f4800 msgr2=0x7f92a41066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:50.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.245+0000 7f92ab621700 1 --2- 172.21.15.174:0/3554189366 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92a40f4800 0x7f92a41066b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f92a0013510 tx=0x7f92a000b040).stop 2023-12-25T21:13:50.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.245+0000 7f92ab621700 1 -- 172.21.15.174:0/3554189366 shutdown_connections 2023-12-25T21:13:50.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.245+0000 7f92ab621700 1 --2- 172.21.15.174:0/3554189366 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f929002ff60 0x7f9290032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:50.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.245+0000 7f92ab621700 1 --2- 172.21.15.174:0/3554189366 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92a40f4800 0x7f92a41066b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:50.245 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.245+0000 7f92ab621700 1 -- 172.21.15.174:0/3554189366 >> 172.21.15.174:0/3554189366 conn(0x7f92a40f0260 msgr2=0x7f92a40f0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:50.246 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.245+0000 7f92ab621700 1 -- 172.21.15.174:0/3554189366 shutdown_connections 2023-12-25T21:13:50.246 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:50.245+0000 7f92ab621700 1 -- 172.21.15.174:0/3554189366 wait complete. 2023-12-25T21:13:50.246 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:50.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:50 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:50.243893+0000 mon.smithi061 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.174:0/3554189366' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:51.626 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:51.626 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:52.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:52 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:50.632994+0000 mgr.smithi061.jxxeke (mgr.14164) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:52.846 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.846+0000 7fbf03fff700 1 -- 172.21.15.174:0/2833573811 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf040f62a0 msgr2=0x7fbf040f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:52.846 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.846+0000 7fbf03fff700 1 --2- 172.21.15.174:0/2833573811 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf040f62a0 0x7fbf040f66c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7fbef8009fb0 tx=0x7fbef8009b30).stop 2023-12-25T21:13:52.846 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.846+0000 7fbf03fff700 1 -- 172.21.15.174:0/2833573811 shutdown_connections 2023-12-25T21:13:52.846 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.846+0000 7fbf03fff700 1 --2- 172.21.15.174:0/2833573811 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf040f62a0 0x7fbf040f66c0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:52.846 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.846+0000 7fbf03fff700 1 -- 172.21.15.174:0/2833573811 >> 172.21.15.174:0/2833573811 conn(0x7fbf040f1aa0 msgr2=0x7fbf040f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:52.846 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.846+0000 7fbf03fff700 1 -- 172.21.15.174:0/2833573811 shutdown_connections 2023-12-25T21:13:52.846 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.846+0000 7fbf03fff700 1 -- 172.21.15.174:0/2833573811 wait complete. 2023-12-25T21:13:52.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.847+0000 7fbf03fff700 1 Processor -- start 2023-12-25T21:13:52.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.847+0000 7fbf03fff700 1 -- start start 2023-12-25T21:13:52.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.847+0000 7fbf03fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf040f62a0 0x7fbf041064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:52.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.847+0000 7fbf03fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf041069a0 con 0x7fbf040f62a0 2023-12-25T21:13:52.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.847+0000 7fbf02ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf040f62a0 0x7fbf041064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:52.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.847+0000 7fbf02ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf040f62a0 0x7fbf041064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:50882/0 (socket says 172.21.15.174:50882) 2023-12-25T21:13:52.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.847+0000 7fbf02ffd700 1 -- 172.21.15.174:0/1815887865 learned_addr learned my addr 172.21.15.174:0/1815887865 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:52.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.847+0000 7fbf02ffd700 1 -- 172.21.15.174:0/1815887865 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef8005040 con 0x7fbf040f62a0 2023-12-25T21:13:52.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.848+0000 7fbf02ffd700 1 --2- 172.21.15.174:0/1815887865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf040f62a0 0x7fbf041064b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fbef8009830 tx=0x7fbef800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:52.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.848+0000 7fbeeb7fe700 1 -- 172.21.15.174:0/1815887865 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbef8009320 con 0x7fbf040f62a0 2023-12-25T21:13:52.848 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.848+0000 7fbf03fff700 1 -- 172.21.15.174:0/1815887865 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf04106ba0 con 0x7fbf040f62a0 2023-12-25T21:13:52.849 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.848+0000 7fbeeb7fe700 1 -- 172.21.15.174:0/1815887865 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbef80136c0 con 0x7fbf040f62a0 2023-12-25T21:13:52.849 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.848+0000 7fbeeb7fe700 1 -- 172.21.15.174:0/1815887865 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbef801ee90 con 0x7fbf040f62a0 2023-12-25T21:13:52.849 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.848+0000 7fbf03fff700 1 -- 172.21.15.174:0/1815887865 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf04107030 con 0x7fbf040f62a0 2023-12-25T21:13:52.850 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.849+0000 7fbeeb7fe700 1 -- 172.21.15.174:0/1815887865 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbef8014440 con 0x7fbf040f62a0 2023-12-25T21:13:52.850 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.849+0000 7fbeeb7fe700 1 --2- 172.21.15.174:0/1815887865 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbeec02ff10 0x7fbeec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:52.850 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.849+0000 7fbf03fff700 1 -- 172.21.15.174:0/1815887865 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf04045ad0 con 0x7fbf040f62a0 2023-12-25T21:13:52.850 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.849+0000 7fbeeb7fe700 1 -- 172.21.15.174:0/1815887865 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbef8041330 con 0x7fbf040f62a0 2023-12-25T21:13:52.850 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.849+0000 7fbf027fc700 1 --2- 172.21.15.174:0/1815887865 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbeec02ff10 0x7fbeec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:52.850 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.849+0000 7fbf027fc700 1 --2- 172.21.15.174:0/1815887865 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbeec02ff10 0x7fbeec0323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fbef40096a0 tx=0x7fbef4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:52.853 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:52.854+0000 7fbeeb7fe700 1 -- 172.21.15.174:0/1815887865 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbef80273f0 con 0x7fbf040f62a0 2023-12-25T21:13:53.056 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:53.056+0000 7fbf03fff700 1 -- 172.21.15.174:0/1815887865 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf04107ee0 con 0x7fbf040f62a0 2023-12-25T21:13:53.056 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:53.057+0000 7fbeeb7fe700 1 -- 172.21.15.174:0/1815887865 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbef802c2d0 con 0x7fbf040f62a0 2023-12-25T21:13:53.057 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:53.057 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:53.058 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:53.058+0000 7fbf03fff700 1 -- 172.21.15.174:0/1815887865 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbeec02ff10 msgr2=0x7fbeec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:53.058 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:53.058+0000 7fbf03fff700 1 --2- 172.21.15.174:0/1815887865 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbeec02ff10 0x7fbeec0323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fbef40096a0 tx=0x7fbef4006b40).stop 2023-12-25T21:13:53.058 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:53.058+0000 7fbf03fff700 1 -- 172.21.15.174:0/1815887865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf040f62a0 msgr2=0x7fbf041064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:53.058 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:53.058+0000 7fbf03fff700 1 --2- 172.21.15.174:0/1815887865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf040f62a0 0x7fbf041064b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fbef8009830 tx=0x7fbef800b040).stop 2023-12-25T21:13:53.058 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:53.058+0000 7fbf03fff700 1 -- 172.21.15.174:0/1815887865 shutdown_connections 2023-12-25T21:13:53.058 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:53.058+0000 7fbf03fff700 1 --2- 172.21.15.174:0/1815887865 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbeec02ff10 0x7fbeec0323d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:53.058 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:53.058+0000 7fbf03fff700 1 --2- 172.21.15.174:0/1815887865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbf040f62a0 0x7fbf041064b0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:53.058 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:53.058+0000 7fbf03fff700 1 -- 172.21.15.174:0/1815887865 >> 172.21.15.174:0/1815887865 conn(0x7fbf040f1aa0 msgr2=0x7fbf040f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:53.059 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:53.058+0000 7fbf03fff700 1 -- 172.21.15.174:0/1815887865 shutdown_connections 2023-12-25T21:13:53.059 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:53.058+0000 7fbf03fff700 1 -- 172.21.15.174:0/1815887865 wait complete. 2023-12-25T21:13:53.059 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:53.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:52.708581+0000 mon.smithi061 (mon.0) 279 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/mirror_snapshot_schedule"}]: dispatch 2023-12-25T21:13:53.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:52.709326+0000 mon.smithi061 (mon.0) 280 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/trash_purge_schedule"}]: dispatch 2023-12-25T21:13:53.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:53.057222+0000 mon.smithi061 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.174:0/1815887865' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:54.451 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:54.451 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:54.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:54 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:52.633313+0000 mgr.smithi061.jxxeke (mgr.14164) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:55.650 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.649+0000 7f826759d700 1 -- 172.21.15.174:0/561287060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82600f51a0 msgr2=0x7f82600f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:55.650 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.649+0000 7f826759d700 1 --2- 172.21.15.174:0/561287060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82600f51a0 0x7f82600f55c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f8250009fb0 tx=0x7f8250009b30).stop 2023-12-25T21:13:55.650 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.649+0000 7f826759d700 1 -- 172.21.15.174:0/561287060 shutdown_connections 2023-12-25T21:13:55.650 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.649+0000 7f826759d700 1 --2- 172.21.15.174:0/561287060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82600f51a0 0x7f82600f55c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:55.650 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.649+0000 7f826759d700 1 -- 172.21.15.174:0/561287060 >> 172.21.15.174:0/561287060 conn(0x7f82600f09a0 msgr2=0x7f82600f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:55.650 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.650+0000 7f826759d700 1 -- 172.21.15.174:0/561287060 shutdown_connections 2023-12-25T21:13:55.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.650+0000 7f826759d700 1 -- 172.21.15.174:0/561287060 wait complete. 2023-12-25T21:13:55.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.650+0000 7f826759d700 1 Processor -- start 2023-12-25T21:13:55.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.650+0000 7f826759d700 1 -- start start 2023-12-25T21:13:55.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.650+0000 7f826759d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82600f51a0 0x7f82601091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:55.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.650+0000 7f826759d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8260060220 con 0x7f82600f51a0 2023-12-25T21:13:55.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.650+0000 7f826659b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82600f51a0 0x7f82601091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:55.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.650+0000 7f826659b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82600f51a0 0x7f82601091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:50900/0 (socket says 172.21.15.174:50900) 2023-12-25T21:13:55.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.650+0000 7f826659b700 1 -- 172.21.15.174:0/3325135726 learned_addr learned my addr 172.21.15.174:0/3325135726 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:55.652 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.651+0000 7f826659b700 1 -- 172.21.15.174:0/3325135726 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8250005040 con 0x7f82600f51a0 2023-12-25T21:13:55.652 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.651+0000 7f826659b700 1 --2- 172.21.15.174:0/3325135726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82600f51a0 0x7f82601091d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f8250009830 tx=0x7f825000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:55.652 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.651+0000 7f8256ffd700 1 -- 172.21.15.174:0/3325135726 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8250009320 con 0x7f82600f51a0 2023-12-25T21:13:55.652 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.651+0000 7f826759d700 1 -- 172.21.15.174:0/3325135726 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8260060420 con 0x7f82600f51a0 2023-12-25T21:13:55.652 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.651+0000 7f8256ffd700 1 -- 172.21.15.174:0/3325135726 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82500136c0 con 0x7f82600f51a0 2023-12-25T21:13:55.652 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.651+0000 7f8256ffd700 1 -- 172.21.15.174:0/3325135726 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f825001ee90 con 0x7f82600f51a0 2023-12-25T21:13:55.652 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.651+0000 7f826759d700 1 -- 172.21.15.174:0/3325135726 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8260109880 con 0x7f82600f51a0 2023-12-25T21:13:55.652 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.652+0000 7f8256ffd700 1 -- 172.21.15.174:0/3325135726 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f825001e420 con 0x7f82600f51a0 2023-12-25T21:13:55.653 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.652+0000 7f8256ffd700 1 --2- 172.21.15.174:0/3325135726 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f824c02ff10 0x7f824c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:55.653 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.652+0000 7f8256ffd700 1 -- 172.21.15.174:0/3325135726 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8250041020 con 0x7f82600f51a0 2023-12-25T21:13:55.653 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.653+0000 7f8265d9a700 1 --2- 172.21.15.174:0/3325135726 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f824c02ff10 0x7f824c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:55.653 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.653+0000 7f826759d700 1 -- 172.21.15.174:0/3325135726 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8260059100 con 0x7f82600f51a0 2023-12-25T21:13:55.653 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.653+0000 7f8265d9a700 1 --2- 172.21.15.174:0/3325135726 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f824c02ff10 0x7f824c0323d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f825c0096a0 tx=0x7f825c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:55.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.658+0000 7f8256ffd700 1 -- 172.21.15.174:0/3325135726 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f825001c790 con 0x7f82600f51a0 2023-12-25T21:13:55.863 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.862+0000 7f826759d700 1 -- 172.21.15.174:0/3325135726 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8260045ad0 con 0x7f82600f51a0 2023-12-25T21:13:55.863 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.864+0000 7f8256ffd700 1 -- 172.21.15.174:0/3325135726 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f825000e2a0 con 0x7f82600f51a0 2023-12-25T21:13:55.864 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:55.864 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:55.865 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.865+0000 7f826759d700 1 -- 172.21.15.174:0/3325135726 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f824c02ff10 msgr2=0x7f824c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:55.866 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.865+0000 7f826759d700 1 --2- 172.21.15.174:0/3325135726 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f824c02ff10 0x7f824c0323d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f825c0096a0 tx=0x7f825c006b40).stop 2023-12-25T21:13:55.866 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.865+0000 7f826759d700 1 -- 172.21.15.174:0/3325135726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82600f51a0 msgr2=0x7f82601091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:55.866 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.865+0000 7f826759d700 1 --2- 172.21.15.174:0/3325135726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82600f51a0 0x7f82601091d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f8250009830 tx=0x7f825000b040).stop 2023-12-25T21:13:55.866 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.865+0000 7f826759d700 1 -- 172.21.15.174:0/3325135726 shutdown_connections 2023-12-25T21:13:55.866 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.865+0000 7f826759d700 1 --2- 172.21.15.174:0/3325135726 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f824c02ff10 0x7f824c0323d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:55.866 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.865+0000 7f826759d700 1 --2- 172.21.15.174:0/3325135726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82600f51a0 0x7f82601091d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:55.866 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.865+0000 7f826759d700 1 -- 172.21.15.174:0/3325135726 >> 172.21.15.174:0/3325135726 conn(0x7f82600f09a0 msgr2=0x7f82600f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:55.866 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.865+0000 7f826759d700 1 -- 172.21.15.174:0/3325135726 shutdown_connections 2023-12-25T21:13:55.866 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:55.865+0000 7f826759d700 1 -- 172.21.15.174:0/3325135726 wait complete. 2023-12-25T21:13:55.866 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:56.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:56 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:54.633661+0000 mgr.smithi061.jxxeke (mgr.14164) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:56.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:56 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:55.864100+0000 mon.smithi061 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.174:0/3325135726' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:13:57.250 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:13:57.250 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:13:58.404 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.402+0000 7f790420d700 1 -- 172.21.15.174:0/3804275158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78fc060220 msgr2=0x7f78fc0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:58.404 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.402+0000 7f790420d700 1 --2- 172.21.15.174:0/3804275158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78fc060220 0x7f78fc0f9150 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f78f8009fb0 tx=0x7f78f8009b30).stop 2023-12-25T21:13:58.404 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.402+0000 7f790420d700 1 -- 172.21.15.174:0/3804275158 shutdown_connections 2023-12-25T21:13:58.404 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.402+0000 7f790420d700 1 --2- 172.21.15.174:0/3804275158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78fc060220 0x7f78fc0f9150 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:58.404 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.402+0000 7f790420d700 1 -- 172.21.15.174:0/3804275158 >> 172.21.15.174:0/3804275158 conn(0x7f78fc0f0b70 msgr2=0x7f78fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:58.404 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.402+0000 7f790420d700 1 -- 172.21.15.174:0/3804275158 shutdown_connections 2023-12-25T21:13:58.404 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.402+0000 7f790420d700 1 -- 172.21.15.174:0/3804275158 wait complete. 2023-12-25T21:13:58.404 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.403+0000 7f790420d700 1 Processor -- start 2023-12-25T21:13:58.404 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.403+0000 7f790420d700 1 -- start start 2023-12-25T21:13:58.404 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.403+0000 7f790420d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78fc060220 0x7f78fc0f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:58.404 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.403+0000 7f790420d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0f5050 con 0x7f78fc060220 2023-12-25T21:13:58.404 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.403+0000 7f7901fa9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78fc060220 0x7f78fc0f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:58.405 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.403+0000 7f7901fa9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78fc060220 0x7f78fc0f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:57762/0 (socket says 172.21.15.174:57762) 2023-12-25T21:13:58.405 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.403+0000 7f7901fa9700 1 -- 172.21.15.174:0/4226119609 learned_addr learned my addr 172.21.15.174:0/4226119609 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:13:58.405 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.404+0000 7f7901fa9700 1 -- 172.21.15.174:0/4226119609 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f8005040 con 0x7f78fc060220 2023-12-25T21:13:58.405 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.404+0000 7f7901fa9700 1 --2- 172.21.15.174:0/4226119609 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78fc060220 0x7f78fc0f6bd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f78f8013510 tx=0x7f78f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:58.405 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.404+0000 7f78f27fc700 1 -- 172.21.15.174:0/4226119609 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78f8009320 con 0x7f78fc060220 2023-12-25T21:13:58.405 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.404+0000 7f78f27fc700 1 -- 172.21.15.174:0/4226119609 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f78f8013f00 con 0x7f78fc060220 2023-12-25T21:13:58.405 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.404+0000 7f790420d700 1 -- 172.21.15.174:0/4226119609 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78fc0f51f0 con 0x7f78fc060220 2023-12-25T21:13:58.405 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.404+0000 7f78f27fc700 1 -- 172.21.15.174:0/4226119609 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78f8014dc0 con 0x7f78fc060220 2023-12-25T21:13:58.405 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.404+0000 7f790420d700 1 -- 172.21.15.174:0/4226119609 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78fc0f5680 con 0x7f78fc060220 2023-12-25T21:13:58.406 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.405+0000 7f78f27fc700 1 -- 172.21.15.174:0/4226119609 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f78f80095f0 con 0x7f78fc060220 2023-12-25T21:13:58.406 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.405+0000 7f790420d700 1 -- 172.21.15.174:0/4226119609 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78fc059100 con 0x7f78fc060220 2023-12-25T21:13:58.406 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.405+0000 7f78f27fc700 1 --2- 172.21.15.174:0/4226119609 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f78e802ff60 0x7f78e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:13:58.406 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.405+0000 7f78f27fc700 1 -- 172.21.15.174:0/4226119609 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f78f80410c0 con 0x7f78fc060220 2023-12-25T21:13:58.406 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.405+0000 7f79017a8700 1 --2- 172.21.15.174:0/4226119609 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f78e802ff60 0x7f78e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:13:58.406 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.406+0000 7f79017a8700 1 --2- 172.21.15.174:0/4226119609 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f78e802ff60 0x7f78e8032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f78ec0096a0 tx=0x7f78ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:13:58.409 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.410+0000 7f78f27fc700 1 -- 172.21.15.174:0/4226119609 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f78f801c790 con 0x7f78fc060220 2023-12-25T21:13:58.619 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.618+0000 7f790420d700 1 -- 172.21.15.174:0/4226119609 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78fc0fbf50 con 0x7f78fc060220 2023-12-25T21:13:58.619 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.619+0000 7f78f27fc700 1 -- 172.21.15.174:0/4226119609 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f78f8010230 con 0x7f78fc060220 2023-12-25T21:13:58.619 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:13:58.619 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:13:58.620 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.621+0000 7f790420d700 1 -- 172.21.15.174:0/4226119609 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f78e802ff60 msgr2=0x7f78e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:58.621 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.621+0000 7f790420d700 1 --2- 172.21.15.174:0/4226119609 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f78e802ff60 0x7f78e8032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f78ec0096a0 tx=0x7f78ec006b40).stop 2023-12-25T21:13:58.621 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.621+0000 7f790420d700 1 -- 172.21.15.174:0/4226119609 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78fc060220 msgr2=0x7f78fc0f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:13:58.621 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.621+0000 7f790420d700 1 --2- 172.21.15.174:0/4226119609 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78fc060220 0x7f78fc0f6bd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f78f8013510 tx=0x7f78f800b040).stop 2023-12-25T21:13:58.621 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.621+0000 7f790420d700 1 -- 172.21.15.174:0/4226119609 shutdown_connections 2023-12-25T21:13:58.621 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.621+0000 7f790420d700 1 --2- 172.21.15.174:0/4226119609 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f78e802ff60 0x7f78e8032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:58.621 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.621+0000 7f790420d700 1 --2- 172.21.15.174:0/4226119609 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78fc060220 0x7f78fc0f6bd0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:13:58.621 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.621+0000 7f790420d700 1 -- 172.21.15.174:0/4226119609 >> 172.21.15.174:0/4226119609 conn(0x7f78fc0f0b70 msgr2=0x7f78fc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:13:58.621 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.621+0000 7f790420d700 1 -- 172.21.15.174:0/4226119609 shutdown_connections 2023-12-25T21:13:58.621 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:13:58.621+0000 7f790420d700 1 -- 172.21.15.174:0/4226119609 wait complete. 2023-12-25T21:13:58.622 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:13:58.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:58 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:56.634030+0000 mgr.smithi061.jxxeke (mgr.14164) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:13:59.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:13:59 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:13:58.619529+0000 mon.smithi061 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.174:0/4226119609' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:00.043 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:00.043 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:00.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:00 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:13:58.634254+0000 mgr.smithi061.jxxeke (mgr.14164) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:01.228 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.227+0000 7f40f5f95700 1 -- 172.21.15.174:0/2077902426 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40f0060280 msgr2=0x7f40f00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:01.228 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.227+0000 7f40f5f95700 1 --2- 172.21.15.174:0/2077902426 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40f0060280 0x7f40f00f90f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f40d8009fb0 tx=0x7f40d8009b30).stop 2023-12-25T21:14:01.228 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.227+0000 7f40f5f95700 1 -- 172.21.15.174:0/2077902426 shutdown_connections 2023-12-25T21:14:01.228 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.227+0000 7f40f5f95700 1 --2- 172.21.15.174:0/2077902426 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40f0060280 0x7f40f00f90f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:01.228 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.227+0000 7f40f5f95700 1 -- 172.21.15.174:0/2077902426 >> 172.21.15.174:0/2077902426 conn(0x7f40f00f0b70 msgr2=0x7f40f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:01.229 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.227+0000 7f40f5f95700 1 -- 172.21.15.174:0/2077902426 shutdown_connections 2023-12-25T21:14:01.229 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.227+0000 7f40f5f95700 1 -- 172.21.15.174:0/2077902426 wait complete. 2023-12-25T21:14:01.229 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.228+0000 7f40f5f95700 1 Processor -- start 2023-12-25T21:14:01.229 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.228+0000 7f40f5f95700 1 -- start start 2023-12-25T21:14:01.229 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.228+0000 7f40f5f95700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40f0060280 0x7f40f01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:01.229 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.228+0000 7f40f5f95700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f0106bd0 con 0x7f40f0060280 2023-12-25T21:14:01.229 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.228+0000 7f40ef7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40f0060280 0x7f40f01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:01.229 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.228+0000 7f40ef7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40f0060280 0x7f40f01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:57776/0 (socket says 172.21.15.174:57776) 2023-12-25T21:14:01.229 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.228+0000 7f40ef7fe700 1 -- 172.21.15.174:0/3240760439 learned_addr learned my addr 172.21.15.174:0/3240760439 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:01.229 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.228+0000 7f40ef7fe700 1 -- 172.21.15.174:0/3240760439 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d8005040 con 0x7f40f0060280 2023-12-25T21:14:01.229 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.229+0000 7f40ef7fe700 1 --2- 172.21.15.174:0/3240760439 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40f0060280 0x7f40f01066e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f40d8013510 tx=0x7f40d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:01.230 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.229+0000 7f40e7fff700 1 -- 172.21.15.174:0/3240760439 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40d8009320 con 0x7f40f0060280 2023-12-25T21:14:01.230 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.229+0000 7f40f5f95700 1 -- 172.21.15.174:0/3240760439 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f0106dd0 con 0x7f40f0060280 2023-12-25T21:14:01.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.229+0000 7f40e7fff700 1 -- 172.21.15.174:0/3240760439 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40d8013f00 con 0x7f40f0060280 2023-12-25T21:14:01.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.229+0000 7f40e7fff700 1 -- 172.21.15.174:0/3240760439 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40d8014dc0 con 0x7f40f0060280 2023-12-25T21:14:01.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.229+0000 7f40f5f95700 1 -- 172.21.15.174:0/3240760439 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f0107260 con 0x7f40f0060280 2023-12-25T21:14:01.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.230+0000 7f40e7fff700 1 -- 172.21.15.174:0/3240760439 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40d80095f0 con 0x7f40f0060280 2023-12-25T21:14:01.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.230+0000 7f40f5f95700 1 -- 172.21.15.174:0/3240760439 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f0045ad0 con 0x7f40f0060280 2023-12-25T21:14:01.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.230+0000 7f40e7fff700 1 --2- 172.21.15.174:0/3240760439 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f40dc02ff60 0x7f40dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:01.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.230+0000 7f40e7fff700 1 -- 172.21.15.174:0/3240760439 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f40d80410c0 con 0x7f40f0060280 2023-12-25T21:14:01.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.230+0000 7f40eeffd700 1 --2- 172.21.15.174:0/3240760439 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f40dc02ff60 0x7f40dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:01.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.230+0000 7f40eeffd700 1 --2- 172.21.15.174:0/3240760439 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f40dc02ff60 0x7f40dc032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f40e00096a0 tx=0x7f40e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:01.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.235+0000 7f40e7fff700 1 -- 172.21.15.174:0/3240760439 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40d8010230 con 0x7f40f0060280 2023-12-25T21:14:01.439 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.440+0000 7f40f5f95700 1 -- 172.21.15.174:0/3240760439 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40f0108110 con 0x7f40f0060280 2023-12-25T21:14:01.440 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.441+0000 7f40e7fff700 1 -- 172.21.15.174:0/3240760439 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f40d8010230 con 0x7f40f0060280 2023-12-25T21:14:01.440 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:01.441 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:01.442 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.442+0000 7f40f5f95700 1 -- 172.21.15.174:0/3240760439 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f40dc02ff60 msgr2=0x7f40dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:01.442 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.442+0000 7f40f5f95700 1 --2- 172.21.15.174:0/3240760439 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f40dc02ff60 0x7f40dc032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f40e00096a0 tx=0x7f40e0006b40).stop 2023-12-25T21:14:01.442 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.442+0000 7f40f5f95700 1 -- 172.21.15.174:0/3240760439 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40f0060280 msgr2=0x7f40f01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:01.442 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.442+0000 7f40f5f95700 1 --2- 172.21.15.174:0/3240760439 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40f0060280 0x7f40f01066e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f40d8013510 tx=0x7f40d800b040).stop 2023-12-25T21:14:01.442 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.442+0000 7f40f5f95700 1 -- 172.21.15.174:0/3240760439 shutdown_connections 2023-12-25T21:14:01.442 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.442+0000 7f40f5f95700 1 --2- 172.21.15.174:0/3240760439 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f40dc02ff60 0x7f40dc032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:01.442 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.442+0000 7f40f5f95700 1 --2- 172.21.15.174:0/3240760439 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40f0060280 0x7f40f01066e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:01.442 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.442+0000 7f40f5f95700 1 -- 172.21.15.174:0/3240760439 >> 172.21.15.174:0/3240760439 conn(0x7f40f00f0b70 msgr2=0x7f40f00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:01.442 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.442+0000 7f40f5f95700 1 -- 172.21.15.174:0/3240760439 shutdown_connections 2023-12-25T21:14:01.442 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:01.442+0000 7f40f5f95700 1 -- 172.21.15.174:0/3240760439 wait complete. 2023-12-25T21:14:01.443 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:02.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:02 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:00.634522+0000 mgr.smithi061.jxxeke (mgr.14164) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:02.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:02 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:01.441117+0000 mon.smithi061 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.174:0/3240760439' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:02.822 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:02.823 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:04.065 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.062+0000 7fad63bf7700 1 -- 172.21.15.174:0/1400493717 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad5c0f5410 msgr2=0x7fad5c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:04.066 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.062+0000 7fad63bf7700 1 --2- 172.21.15.174:0/1400493717 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad5c0f5410 0x7fad5c0f5830 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fad4c009fb0 tx=0x7fad4c009b30).stop 2023-12-25T21:14:04.066 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.062+0000 7fad63bf7700 1 -- 172.21.15.174:0/1400493717 shutdown_connections 2023-12-25T21:14:04.066 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.062+0000 7fad63bf7700 1 --2- 172.21.15.174:0/1400493717 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad5c0f5410 0x7fad5c0f5830 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:04.066 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.062+0000 7fad63bf7700 1 -- 172.21.15.174:0/1400493717 >> 172.21.15.174:0/1400493717 conn(0x7fad5c0f0b70 msgr2=0x7fad5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:04.066 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.063+0000 7fad63bf7700 1 -- 172.21.15.174:0/1400493717 shutdown_connections 2023-12-25T21:14:04.066 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.063+0000 7fad63bf7700 1 -- 172.21.15.174:0/1400493717 wait complete. 2023-12-25T21:14:04.066 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.063+0000 7fad63bf7700 1 Processor -- start 2023-12-25T21:14:04.067 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.063+0000 7fad63bf7700 1 -- start start 2023-12-25T21:14:04.067 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.063+0000 7fad63bf7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad5c0f5410 0x7fad5c105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:04.067 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.063+0000 7fad63bf7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c105af0 con 0x7fad5c0f5410 2023-12-25T21:14:04.067 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.064+0000 7fad61993700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad5c0f5410 0x7fad5c105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:04.067 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.064+0000 7fad61993700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad5c0f5410 0x7fad5c105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:57786/0 (socket says 172.21.15.174:57786) 2023-12-25T21:14:04.067 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.064+0000 7fad61993700 1 -- 172.21.15.174:0/3054982125 learned_addr learned my addr 172.21.15.174:0/3054982125 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:04.067 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.064+0000 7fad61993700 1 -- 172.21.15.174:0/3054982125 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c005040 con 0x7fad5c0f5410 2023-12-25T21:14:04.067 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.064+0000 7fad61993700 1 --2- 172.21.15.174:0/3054982125 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad5c0f5410 0x7fad5c105600 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fad4c013510 tx=0x7fad4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:04.067 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.064+0000 7fad527fc700 1 -- 172.21.15.174:0/3054982125 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad4c009320 con 0x7fad5c0f5410 2023-12-25T21:14:04.067 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.064+0000 7fad63bf7700 1 -- 172.21.15.174:0/3054982125 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5c105cf0 con 0x7fad5c0f5410 2023-12-25T21:14:04.067 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.064+0000 7fad527fc700 1 -- 172.21.15.174:0/3054982125 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad4c013f00 con 0x7fad5c0f5410 2023-12-25T21:14:04.068 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.064+0000 7fad527fc700 1 -- 172.21.15.174:0/3054982125 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad4c014d90 con 0x7fad5c0f5410 2023-12-25T21:14:04.068 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.064+0000 7fad63bf7700 1 -- 172.21.15.174:0/3054982125 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5c106180 con 0x7fad5c0f5410 2023-12-25T21:14:04.069 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.065+0000 7fad527fc700 1 -- 172.21.15.174:0/3054982125 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad4c0095f0 con 0x7fad5c0f5410 2023-12-25T21:14:04.069 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.065+0000 7fad63bf7700 1 -- 172.21.15.174:0/3054982125 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad5c045ad0 con 0x7fad5c0f5410 2023-12-25T21:14:04.069 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.065+0000 7fad527fc700 1 --2- 172.21.15.174:0/3054982125 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fad4802ff60 0x7fad48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:04.069 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.065+0000 7fad527fc700 1 -- 172.21.15.174:0/3054982125 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad4c041080 con 0x7fad5c0f5410 2023-12-25T21:14:04.069 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.066+0000 7fad61192700 1 --2- 172.21.15.174:0/3054982125 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fad4802ff60 0x7fad48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:04.069 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.066+0000 7fad61192700 1 --2- 172.21.15.174:0/3054982125 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fad4802ff60 0x7fad48032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fad580096a0 tx=0x7fad58006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:04.071 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.070+0000 7fad527fc700 1 -- 172.21.15.174:0/3054982125 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad4c014420 con 0x7fad5c0f5410 2023-12-25T21:14:04.274 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.274+0000 7fad63bf7700 1 -- 172.21.15.174:0/3054982125 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad5c0fae60 con 0x7fad5c0f5410 2023-12-25T21:14:04.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.274+0000 7fad527fc700 1 -- 172.21.15.174:0/3054982125 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad4c010230 con 0x7fad5c0f5410 2023-12-25T21:14:04.275 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:04.275 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:04.277 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.276+0000 7fad63bf7700 1 -- 172.21.15.174:0/3054982125 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fad4802ff60 msgr2=0x7fad48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:04.277 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.276+0000 7fad63bf7700 1 --2- 172.21.15.174:0/3054982125 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fad4802ff60 0x7fad48032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fad580096a0 tx=0x7fad58006b40).stop 2023-12-25T21:14:04.277 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.276+0000 7fad63bf7700 1 -- 172.21.15.174:0/3054982125 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad5c0f5410 msgr2=0x7fad5c105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:04.277 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.276+0000 7fad63bf7700 1 --2- 172.21.15.174:0/3054982125 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad5c0f5410 0x7fad5c105600 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fad4c013510 tx=0x7fad4c00b040).stop 2023-12-25T21:14:04.277 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.276+0000 7fad63bf7700 1 -- 172.21.15.174:0/3054982125 shutdown_connections 2023-12-25T21:14:04.277 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.276+0000 7fad63bf7700 1 --2- 172.21.15.174:0/3054982125 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fad4802ff60 0x7fad48032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:04.277 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.276+0000 7fad63bf7700 1 --2- 172.21.15.174:0/3054982125 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad5c0f5410 0x7fad5c105600 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:04.277 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.276+0000 7fad63bf7700 1 -- 172.21.15.174:0/3054982125 >> 172.21.15.174:0/3054982125 conn(0x7fad5c0f0b70 msgr2=0x7fad5c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:04.278 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.276+0000 7fad63bf7700 1 -- 172.21.15.174:0/3054982125 shutdown_connections 2023-12-25T21:14:04.278 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:04.276+0000 7fad63bf7700 1 -- 172.21.15.174:0/3054982125 wait complete. 2023-12-25T21:14:04.278 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:04.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:04 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:02.634866+0000 mgr.smithi061.jxxeke (mgr.14164) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:04.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:04 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:04.275972+0000 mon.smithi061 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.174:0/3054982125' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:05.647 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:05.647 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:06.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:06 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:04.635233+0000 mgr.smithi061.jxxeke (mgr.14164) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:06.828 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.826+0000 7ffae73e3700 1 -- 172.21.15.174:0/4062726404 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae0060280 msgr2=0x7ffae00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:06.829 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.826+0000 7ffae73e3700 1 --2- 172.21.15.174:0/4062726404 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae0060280 0x7ffae00f90f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7ffad0009fb0 tx=0x7ffad0009b30).stop 2023-12-25T21:14:06.829 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.826+0000 7ffae73e3700 1 -- 172.21.15.174:0/4062726404 shutdown_connections 2023-12-25T21:14:06.829 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.826+0000 7ffae73e3700 1 --2- 172.21.15.174:0/4062726404 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae0060280 0x7ffae00f90f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:06.829 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.826+0000 7ffae73e3700 1 -- 172.21.15.174:0/4062726404 >> 172.21.15.174:0/4062726404 conn(0x7ffae00f0b70 msgr2=0x7ffae00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:06.829 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.826+0000 7ffae73e3700 1 -- 172.21.15.174:0/4062726404 shutdown_connections 2023-12-25T21:14:06.829 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.826+0000 7ffae73e3700 1 -- 172.21.15.174:0/4062726404 wait complete. 2023-12-25T21:14:06.829 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.827+0000 7ffae73e3700 1 Processor -- start 2023-12-25T21:14:06.829 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.827+0000 7ffae73e3700 1 -- start start 2023-12-25T21:14:06.829 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.827+0000 7ffae73e3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae0060280 0x7ffae01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:06.829 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.827+0000 7ffae73e3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae0106bd0 con 0x7ffae0060280 2023-12-25T21:14:06.829 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.827+0000 7ffae517f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae0060280 0x7ffae01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:06.830 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.827+0000 7ffae517f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae0060280 0x7ffae01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:59442/0 (socket says 172.21.15.174:59442) 2023-12-25T21:14:06.830 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.827+0000 7ffae517f700 1 -- 172.21.15.174:0/1520258333 learned_addr learned my addr 172.21.15.174:0/1520258333 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:06.830 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.828+0000 7ffae517f700 1 -- 172.21.15.174:0/1520258333 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad0005040 con 0x7ffae0060280 2023-12-25T21:14:06.830 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.828+0000 7ffae517f700 1 --2- 172.21.15.174:0/1520258333 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae0060280 0x7ffae01066e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7ffad0013510 tx=0x7ffad000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:06.830 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.828+0000 7ffad5ffb700 1 -- 172.21.15.174:0/1520258333 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffad0009320 con 0x7ffae0060280 2023-12-25T21:14:06.830 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.828+0000 7ffae73e3700 1 -- 172.21.15.174:0/1520258333 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae0106dd0 con 0x7ffae0060280 2023-12-25T21:14:06.830 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.828+0000 7ffad5ffb700 1 -- 172.21.15.174:0/1520258333 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffad0013f00 con 0x7ffae0060280 2023-12-25T21:14:06.830 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.828+0000 7ffad5ffb700 1 -- 172.21.15.174:0/1520258333 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffad0014dc0 con 0x7ffae0060280 2023-12-25T21:14:06.830 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.828+0000 7ffae73e3700 1 -- 172.21.15.174:0/1520258333 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae0107260 con 0x7ffae0060280 2023-12-25T21:14:06.831 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.829+0000 7ffad5ffb700 1 -- 172.21.15.174:0/1520258333 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffad0014420 con 0x7ffae0060280 2023-12-25T21:14:06.831 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.829+0000 7ffae73e3700 1 -- 172.21.15.174:0/1520258333 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae0045ad0 con 0x7ffae0060280 2023-12-25T21:14:06.831 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.829+0000 7ffad5ffb700 1 --2- 172.21.15.174:0/1520258333 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ffacc02ff60 0x7ffacc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:06.831 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.829+0000 7ffad5ffb700 1 -- 172.21.15.174:0/1520258333 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffad0041020 con 0x7ffae0060280 2023-12-25T21:14:06.831 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.829+0000 7ffae497e700 1 --2- 172.21.15.174:0/1520258333 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ffacc02ff60 0x7ffacc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:06.831 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.830+0000 7ffae497e700 1 --2- 172.21.15.174:0/1520258333 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ffacc02ff60 0x7ffacc032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7ffadc0096a0 tx=0x7ffadc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:06.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:06.834+0000 7ffad5ffb700 1 -- 172.21.15.174:0/1520258333 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffad0012330 con 0x7ffae0060280 2023-12-25T21:14:07.044 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:07.044+0000 7ffae73e3700 1 -- 172.21.15.174:0/1520258333 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffae0108090 con 0x7ffae0060280 2023-12-25T21:14:07.045 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:07.045+0000 7ffad5ffb700 1 -- 172.21.15.174:0/1520258333 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffad001cb10 con 0x7ffae0060280 2023-12-25T21:14:07.045 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:07.046 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:07.047 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:07.046+0000 7ffae73e3700 1 -- 172.21.15.174:0/1520258333 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ffacc02ff60 msgr2=0x7ffacc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:07.047 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:07.046+0000 7ffae73e3700 1 --2- 172.21.15.174:0/1520258333 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ffacc02ff60 0x7ffacc032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7ffadc0096a0 tx=0x7ffadc006b40).stop 2023-12-25T21:14:07.047 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:07.046+0000 7ffae73e3700 1 -- 172.21.15.174:0/1520258333 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae0060280 msgr2=0x7ffae01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:07.047 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:07.046+0000 7ffae73e3700 1 --2- 172.21.15.174:0/1520258333 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae0060280 0x7ffae01066e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7ffad0013510 tx=0x7ffad000b040).stop 2023-12-25T21:14:07.047 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:07.046+0000 7ffae73e3700 1 -- 172.21.15.174:0/1520258333 shutdown_connections 2023-12-25T21:14:07.047 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:07.046+0000 7ffae73e3700 1 --2- 172.21.15.174:0/1520258333 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ffacc02ff60 0x7ffacc032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:07.047 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:07.046+0000 7ffae73e3700 1 --2- 172.21.15.174:0/1520258333 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae0060280 0x7ffae01066e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:07.047 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:07.046+0000 7ffae73e3700 1 -- 172.21.15.174:0/1520258333 >> 172.21.15.174:0/1520258333 conn(0x7ffae00f0b70 msgr2=0x7ffae00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:07.048 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:07.046+0000 7ffae73e3700 1 -- 172.21.15.174:0/1520258333 shutdown_connections 2023-12-25T21:14:07.048 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:07.046+0000 7ffae73e3700 1 -- 172.21.15.174:0/1520258333 wait complete. 2023-12-25T21:14:07.048 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:07.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:07 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:07.046139+0000 mon.smithi061 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.174:0/1520258333' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:08.432 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:08.432 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:08.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:08 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:06.635620+0000 mgr.smithi061.jxxeke (mgr.14164) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:09.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.655+0000 7f82e629f700 1 -- 172.21.15.174:0/673261811 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82e0060200 msgr2=0x7f82e00f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:09.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.655+0000 7f82e629f700 1 --2- 172.21.15.174:0/673261811 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82e0060200 0x7f82e00f9170 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f82d0009fb0 tx=0x7f82d0009b30).stop 2023-12-25T21:14:09.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.655+0000 7f82e629f700 1 -- 172.21.15.174:0/673261811 shutdown_connections 2023-12-25T21:14:09.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.655+0000 7f82e629f700 1 --2- 172.21.15.174:0/673261811 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82e0060200 0x7f82e00f9170 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:09.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.655+0000 7f82e629f700 1 -- 172.21.15.174:0/673261811 >> 172.21.15.174:0/673261811 conn(0x7f82e00f0b70 msgr2=0x7f82e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:09.658 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.655+0000 7f82e629f700 1 -- 172.21.15.174:0/673261811 shutdown_connections 2023-12-25T21:14:09.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.655+0000 7f82e629f700 1 -- 172.21.15.174:0/673261811 wait complete. 2023-12-25T21:14:09.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.656+0000 7f82e629f700 1 Processor -- start 2023-12-25T21:14:09.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.656+0000 7f82e629f700 1 -- start start 2023-12-25T21:14:09.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.656+0000 7f82e629f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82e0060200 0x7f82e01066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:09.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.656+0000 7f82e629f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e0106b90 con 0x7f82e0060200 2023-12-25T21:14:09.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.657+0000 7f82dffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82e0060200 0x7f82e01066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:09.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.657+0000 7f82dffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82e0060200 0x7f82e01066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:59466/0 (socket says 172.21.15.174:59466) 2023-12-25T21:14:09.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.657+0000 7f82dffff700 1 -- 172.21.15.174:0/1861140858 learned_addr learned my addr 172.21.15.174:0/1861140858 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:09.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.657+0000 7f82dffff700 1 -- 172.21.15.174:0/1861140858 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d0005040 con 0x7f82e0060200 2023-12-25T21:14:09.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.657+0000 7f82dffff700 1 --2- 172.21.15.174:0/1861140858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82e0060200 0x7f82e01066a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f82d0013510 tx=0x7f82d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:09.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.657+0000 7f82dcff9700 1 -- 172.21.15.174:0/1861140858 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82d0009320 con 0x7f82e0060200 2023-12-25T21:14:09.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.657+0000 7f82e629f700 1 -- 172.21.15.174:0/1861140858 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e0106d90 con 0x7f82e0060200 2023-12-25T21:14:09.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.657+0000 7f82dcff9700 1 -- 172.21.15.174:0/1861140858 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82d0013f00 con 0x7f82e0060200 2023-12-25T21:14:09.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.657+0000 7f82dcff9700 1 -- 172.21.15.174:0/1861140858 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82d0014dc0 con 0x7f82e0060200 2023-12-25T21:14:09.659 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.657+0000 7f82e629f700 1 -- 172.21.15.174:0/1861140858 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e0107220 con 0x7f82e0060200 2023-12-25T21:14:09.660 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.658+0000 7f82dcff9700 1 -- 172.21.15.174:0/1861140858 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f82d00095f0 con 0x7f82e0060200 2023-12-25T21:14:09.660 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.658+0000 7f82e629f700 1 -- 172.21.15.174:0/1861140858 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e0045ad0 con 0x7f82e0060200 2023-12-25T21:14:09.660 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.658+0000 7f82dcff9700 1 --2- 172.21.15.174:0/1861140858 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f82c802ff60 0x7f82c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:09.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.658+0000 7f82dcff9700 1 -- 172.21.15.174:0/1861140858 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f82d00410f0 con 0x7f82e0060200 2023-12-25T21:14:09.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.658+0000 7f82df7fe700 1 --2- 172.21.15.174:0/1861140858 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f82c802ff60 0x7f82c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:09.661 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.659+0000 7f82df7fe700 1 --2- 172.21.15.174:0/1861140858 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f82c802ff60 0x7f82c8032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f82d40096a0 tx=0x7f82d4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:09.664 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.663+0000 7f82dcff9700 1 -- 172.21.15.174:0/1861140858 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f82d0014420 con 0x7f82e0060200 2023-12-25T21:14:09.865 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.865+0000 7f82e629f700 1 -- 172.21.15.174:0/1861140858 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f82e01080d0 con 0x7f82e0060200 2023-12-25T21:14:09.866 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.866+0000 7f82dcff9700 1 -- 172.21.15.174:0/1861140858 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f82d0010230 con 0x7f82e0060200 2023-12-25T21:14:09.866 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:09.867 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:09.868 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.867+0000 7f82e629f700 1 -- 172.21.15.174:0/1861140858 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f82c802ff60 msgr2=0x7f82c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:09.868 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.867+0000 7f82e629f700 1 --2- 172.21.15.174:0/1861140858 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f82c802ff60 0x7f82c8032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f82d40096a0 tx=0x7f82d4006b40).stop 2023-12-25T21:14:09.868 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.867+0000 7f82e629f700 1 -- 172.21.15.174:0/1861140858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82e0060200 msgr2=0x7f82e01066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:09.868 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.867+0000 7f82e629f700 1 --2- 172.21.15.174:0/1861140858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82e0060200 0x7f82e01066a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f82d0013510 tx=0x7f82d000b040).stop 2023-12-25T21:14:09.868 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.867+0000 7f82e629f700 1 -- 172.21.15.174:0/1861140858 shutdown_connections 2023-12-25T21:14:09.868 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.867+0000 7f82e629f700 1 --2- 172.21.15.174:0/1861140858 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f82c802ff60 0x7f82c8032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:09.868 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.867+0000 7f82e629f700 1 --2- 172.21.15.174:0/1861140858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82e0060200 0x7f82e01066a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:09.869 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.867+0000 7f82e629f700 1 -- 172.21.15.174:0/1861140858 >> 172.21.15.174:0/1861140858 conn(0x7f82e00f0b70 msgr2=0x7f82e00f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:09.869 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.867+0000 7f82e629f700 1 -- 172.21.15.174:0/1861140858 shutdown_connections 2023-12-25T21:14:09.869 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:09.867+0000 7f82e629f700 1 -- 172.21.15.174:0/1861140858 wait complete. 2023-12-25T21:14:09.869 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:10.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:10 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:08.635990+0000 mgr.smithi061.jxxeke (mgr.14164) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:10.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:10 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:09.867083+0000 mon.smithi061 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.174:0/1861140858' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:11.278 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:11.278 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:12.501 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.499+0000 7f3cddee5700 1 -- 172.21.15.174:0/1609140212 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cd80f62a0 msgr2=0x7f3cd80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:12.501 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.500+0000 7f3cddee5700 1 --2- 172.21.15.174:0/1609140212 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cd80f62a0 0x7f3cd80f66c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f3cd0009960 tx=0x7f3cd0004d10).stop 2023-12-25T21:14:12.501 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.500+0000 7f3cddee5700 1 -- 172.21.15.174:0/1609140212 shutdown_connections 2023-12-25T21:14:12.501 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.500+0000 7f3cddee5700 1 --2- 172.21.15.174:0/1609140212 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cd80f62a0 0x7f3cd80f66c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:12.501 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.500+0000 7f3cddee5700 1 -- 172.21.15.174:0/1609140212 >> 172.21.15.174:0/1609140212 conn(0x7f3cd80f1ac0 msgr2=0x7f3cd80f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:12.502 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.500+0000 7f3cddee5700 1 -- 172.21.15.174:0/1609140212 shutdown_connections 2023-12-25T21:14:12.502 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.500+0000 7f3cddee5700 1 -- 172.21.15.174:0/1609140212 wait complete. 2023-12-25T21:14:12.502 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.500+0000 7f3cddee5700 1 Processor -- start 2023-12-25T21:14:12.502 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.500+0000 7f3cddee5700 1 -- start start 2023-12-25T21:14:12.502 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.500+0000 7f3cddee5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cd80f62a0 0x7f3cd8106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:12.502 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.500+0000 7f3cddee5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd81069f0 con 0x7f3cd80f62a0 2023-12-25T21:14:12.503 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.501+0000 7f3cdcee3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cd80f62a0 0x7f3cd8106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:12.504 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.501+0000 7f3cdcee3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cd80f62a0 0x7f3cd8106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:59494/0 (socket says 172.21.15.174:59494) 2023-12-25T21:14:12.504 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.501+0000 7f3cdcee3700 1 -- 172.21.15.174:0/72124383 learned_addr learned my addr 172.21.15.174:0/72124383 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:12.504 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.501+0000 7f3cdcee3700 1 -- 172.21.15.174:0/72124383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd0009640 con 0x7f3cd80f62a0 2023-12-25T21:14:12.504 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.501+0000 7f3cdcee3700 1 --2- 172.21.15.174:0/72124383 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cd80f62a0 0x7f3cd8106500 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f3cd0007bd0 tx=0x7f3cd0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:12.504 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.501+0000 7f3cd57fa700 1 -- 172.21.15.174:0/72124383 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cd0018070 con 0x7f3cd80f62a0 2023-12-25T21:14:12.504 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.501+0000 7f3cddee5700 1 -- 172.21.15.174:0/72124383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd8106bf0 con 0x7f3cd80f62a0 2023-12-25T21:14:12.504 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.501+0000 7f3cd57fa700 1 -- 172.21.15.174:0/72124383 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3cd000b870 con 0x7f3cd80f62a0 2023-12-25T21:14:12.504 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.501+0000 7f3cd57fa700 1 -- 172.21.15.174:0/72124383 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cd001d420 con 0x7f3cd80f62a0 2023-12-25T21:14:12.505 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.501+0000 7f3cddee5700 1 -- 172.21.15.174:0/72124383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd8107080 con 0x7f3cd80f62a0 2023-12-25T21:14:12.505 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.502+0000 7f3cd57fa700 1 -- 172.21.15.174:0/72124383 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3cd0024060 con 0x7f3cd80f62a0 2023-12-25T21:14:12.505 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.502+0000 7f3cddee5700 1 -- 172.21.15.174:0/72124383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd8045ad0 con 0x7f3cd80f62a0 2023-12-25T21:14:12.505 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.502+0000 7f3cd57fa700 1 --2- 172.21.15.174:0/72124383 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3cc002ff60 0x7f3cc0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:12.505 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.502+0000 7f3cd57fa700 1 -- 172.21.15.174:0/72124383 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3cd0041150 con 0x7f3cd80f62a0 2023-12-25T21:14:12.505 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.503+0000 7f3cd7fff700 1 --2- 172.21.15.174:0/72124383 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3cc002ff60 0x7f3cc0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:12.505 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.503+0000 7f3cd7fff700 1 --2- 172.21.15.174:0/72124383 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3cc002ff60 0x7f3cc0032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f3ccc0096a0 tx=0x7f3ccc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:12.508 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.507+0000 7f3cd57fa700 1 -- 172.21.15.174:0/72124383 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3cd0025520 con 0x7f3cd80f62a0 2023-12-25T21:14:12.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:12 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:10.636312+0000 mgr.smithi061.jxxeke (mgr.14164) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:12.713 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.712+0000 7f3cddee5700 1 -- 172.21.15.174:0/72124383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3cd8107ef0 con 0x7f3cd80f62a0 2023-12-25T21:14:12.713 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.713+0000 7f3cd57fa700 1 -- 172.21.15.174:0/72124383 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3cd001a2d0 con 0x7f3cd80f62a0 2023-12-25T21:14:12.714 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:12.714 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:12.716 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.715+0000 7f3cddee5700 1 -- 172.21.15.174:0/72124383 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3cc002ff60 msgr2=0x7f3cc0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:12.716 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.715+0000 7f3cddee5700 1 --2- 172.21.15.174:0/72124383 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3cc002ff60 0x7f3cc0032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f3ccc0096a0 tx=0x7f3ccc006b40).stop 2023-12-25T21:14:12.716 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.715+0000 7f3cddee5700 1 -- 172.21.15.174:0/72124383 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cd80f62a0 msgr2=0x7f3cd8106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:12.716 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.715+0000 7f3cddee5700 1 --2- 172.21.15.174:0/72124383 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cd80f62a0 0x7f3cd8106500 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f3cd0007bd0 tx=0x7f3cd0004d10).stop 2023-12-25T21:14:12.716 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.715+0000 7f3cddee5700 1 -- 172.21.15.174:0/72124383 shutdown_connections 2023-12-25T21:14:12.716 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.715+0000 7f3cddee5700 1 --2- 172.21.15.174:0/72124383 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3cc002ff60 0x7f3cc0032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:12.716 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.715+0000 7f3cddee5700 1 --2- 172.21.15.174:0/72124383 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cd80f62a0 0x7f3cd8106500 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:12.716 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.715+0000 7f3cddee5700 1 -- 172.21.15.174:0/72124383 >> 172.21.15.174:0/72124383 conn(0x7f3cd80f1ac0 msgr2=0x7f3cd80f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:12.716 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.715+0000 7f3cddee5700 1 -- 172.21.15.174:0/72124383 shutdown_connections 2023-12-25T21:14:12.716 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:12.715+0000 7f3cddee5700 1 -- 172.21.15.174:0/72124383 wait complete. 2023-12-25T21:14:12.717 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:13.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:13 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:12.714558+0000 mon.smithi061 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.174:0/72124383' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:14.134 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:14.135 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:14.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:14 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:12.636677+0000 mgr.smithi061.jxxeke (mgr.14164) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:15.298 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.296+0000 7f3c2188e700 1 -- 172.21.15.174:0/1746679391 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c0f6510 msgr2=0x7f3c1c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:15.298 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.296+0000 7f3c2188e700 1 --2- 172.21.15.174:0/1746679391 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c0f6510 0x7f3c1c0f6930 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f3c10009fb0 tx=0x7f3c10009b30).stop 2023-12-25T21:14:15.298 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.296+0000 7f3c2188e700 1 -- 172.21.15.174:0/1746679391 shutdown_connections 2023-12-25T21:14:15.298 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.296+0000 7f3c2188e700 1 --2- 172.21.15.174:0/1746679391 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c0f6510 0x7f3c1c0f6930 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:15.298 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.296+0000 7f3c2188e700 1 -- 172.21.15.174:0/1746679391 >> 172.21.15.174:0/1746679391 conn(0x7f3c1c0f1c70 msgr2=0x7f3c1c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:15.298 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.296+0000 7f3c2188e700 1 -- 172.21.15.174:0/1746679391 shutdown_connections 2023-12-25T21:14:15.298 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.296+0000 7f3c2188e700 1 -- 172.21.15.174:0/1746679391 wait complete. 2023-12-25T21:14:15.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.297+0000 7f3c2188e700 1 Processor -- start 2023-12-25T21:14:15.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.297+0000 7f3c2188e700 1 -- start start 2023-12-25T21:14:15.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.297+0000 7f3c2188e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c0f6510 0x7f3c1c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:15.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.297+0000 7f3c2188e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c106c20 con 0x7f3c1c0f6510 2023-12-25T21:14:15.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.297+0000 7f3c1affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c0f6510 0x7f3c1c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:15.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.297+0000 7f3c1affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c0f6510 0x7f3c1c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:59514/0 (socket says 172.21.15.174:59514) 2023-12-25T21:14:15.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.297+0000 7f3c1affd700 1 -- 172.21.15.174:0/970504135 learned_addr learned my addr 172.21.15.174:0/970504135 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:15.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.297+0000 7f3c1affd700 1 -- 172.21.15.174:0/970504135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c10005040 con 0x7f3c1c0f6510 2023-12-25T21:14:15.300 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.298+0000 7f3c1affd700 1 --2- 172.21.15.174:0/970504135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c0f6510 0x7f3c1c106730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f3c10013510 tx=0x7f3c1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:15.300 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.298+0000 7f3c037fe700 1 -- 172.21.15.174:0/970504135 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c10009320 con 0x7f3c1c0f6510 2023-12-25T21:14:15.301 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.298+0000 7f3c2188e700 1 -- 172.21.15.174:0/970504135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c1c106e20 con 0x7f3c1c0f6510 2023-12-25T21:14:15.301 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.298+0000 7f3c037fe700 1 -- 172.21.15.174:0/970504135 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c10013f00 con 0x7f3c1c0f6510 2023-12-25T21:14:15.301 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.298+0000 7f3c037fe700 1 -- 172.21.15.174:0/970504135 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c10014d90 con 0x7f3c1c0f6510 2023-12-25T21:14:15.301 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.298+0000 7f3c2188e700 1 -- 172.21.15.174:0/970504135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c1c1072b0 con 0x7f3c1c0f6510 2023-12-25T21:14:15.301 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.298+0000 7f3c037fe700 1 -- 172.21.15.174:0/970504135 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3c100095f0 con 0x7f3c1c0f6510 2023-12-25T21:14:15.301 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.299+0000 7f3c2188e700 1 -- 172.21.15.174:0/970504135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c1c045ad0 con 0x7f3c1c0f6510 2023-12-25T21:14:15.301 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.299+0000 7f3c037fe700 1 --2- 172.21.15.174:0/970504135 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3c0402ff60 0x7f3c04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:15.301 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.299+0000 7f3c037fe700 1 -- 172.21.15.174:0/970504135 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3c10041080 con 0x7f3c1c0f6510 2023-12-25T21:14:15.301 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.299+0000 7f3c1a7fc700 1 --2- 172.21.15.174:0/970504135 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3c0402ff60 0x7f3c04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:15.301 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.299+0000 7f3c1a7fc700 1 --2- 172.21.15.174:0/970504135 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3c0402ff60 0x7f3c04032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f3c0c0096a0 tx=0x7f3c0c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:15.304 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.304+0000 7f3c037fe700 1 -- 172.21.15.174:0/970504135 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c10014420 con 0x7f3c1c0f6510 2023-12-25T21:14:15.512 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.511+0000 7f3c2188e700 1 -- 172.21.15.174:0/970504135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c1c0fbf60 con 0x7f3c1c0f6510 2023-12-25T21:14:15.513 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.512+0000 7f3c037fe700 1 -- 172.21.15.174:0/970504135 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3c10010230 con 0x7f3c1c0f6510 2023-12-25T21:14:15.513 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:15.513 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:15.514 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.513+0000 7f3c2188e700 1 -- 172.21.15.174:0/970504135 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3c0402ff60 msgr2=0x7f3c04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:15.514 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.513+0000 7f3c2188e700 1 --2- 172.21.15.174:0/970504135 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3c0402ff60 0x7f3c04032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f3c0c0096a0 tx=0x7f3c0c006b40).stop 2023-12-25T21:14:15.515 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.513+0000 7f3c2188e700 1 -- 172.21.15.174:0/970504135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c0f6510 msgr2=0x7f3c1c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:15.515 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.513+0000 7f3c2188e700 1 --2- 172.21.15.174:0/970504135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c0f6510 0x7f3c1c106730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f3c10013510 tx=0x7f3c1000b040).stop 2023-12-25T21:14:15.515 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.514+0000 7f3c2188e700 1 -- 172.21.15.174:0/970504135 shutdown_connections 2023-12-25T21:14:15.515 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.514+0000 7f3c2188e700 1 --2- 172.21.15.174:0/970504135 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3c0402ff60 0x7f3c04032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:15.515 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.514+0000 7f3c2188e700 1 --2- 172.21.15.174:0/970504135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c0f6510 0x7f3c1c106730 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:15.515 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.514+0000 7f3c2188e700 1 -- 172.21.15.174:0/970504135 >> 172.21.15.174:0/970504135 conn(0x7f3c1c0f1c70 msgr2=0x7f3c1c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:15.515 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.514+0000 7f3c2188e700 1 -- 172.21.15.174:0/970504135 shutdown_connections 2023-12-25T21:14:15.515 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:15.514+0000 7f3c2188e700 1 -- 172.21.15.174:0/970504135 wait complete. 2023-12-25T21:14:15.515 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:16.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:16 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:14.636986+0000 mgr.smithi061.jxxeke (mgr.14164) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:16.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:16 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:15.513513+0000 mon.smithi061 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.174:0/970504135' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:16.886 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:16.887 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:18.109 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.107+0000 7f8ec6f86700 1 -- 172.21.15.174:0/3908491190 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec00f5410 msgr2=0x7f8ec00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:18.109 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.107+0000 7f8ec6f86700 1 --2- 172.21.15.174:0/3908491190 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec00f5410 0x7f8ec00f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f8eb4009fb0 tx=0x7f8eb4009b30).stop 2023-12-25T21:14:18.109 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.107+0000 7f8ec6f86700 1 -- 172.21.15.174:0/3908491190 shutdown_connections 2023-12-25T21:14:18.109 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.107+0000 7f8ec6f86700 1 --2- 172.21.15.174:0/3908491190 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec00f5410 0x7f8ec00f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:18.110 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.107+0000 7f8ec6f86700 1 -- 172.21.15.174:0/3908491190 >> 172.21.15.174:0/3908491190 conn(0x7f8ec00f0b70 msgr2=0x7f8ec00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:18.110 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.108+0000 7f8ec6f86700 1 -- 172.21.15.174:0/3908491190 shutdown_connections 2023-12-25T21:14:18.110 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.108+0000 7f8ec6f86700 1 -- 172.21.15.174:0/3908491190 wait complete. 2023-12-25T21:14:18.110 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.108+0000 7f8ec6f86700 1 Processor -- start 2023-12-25T21:14:18.110 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.108+0000 7f8ec6f86700 1 -- start start 2023-12-25T21:14:18.110 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.108+0000 7f8ec6f86700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec00f5410 0x7f8ec0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:18.110 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.108+0000 7f8ec6f86700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec0105b20 con 0x7f8ec00f5410 2023-12-25T21:14:18.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.109+0000 7f8ec4d22700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec00f5410 0x7f8ec0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:18.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.109+0000 7f8ec4d22700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec00f5410 0x7f8ec0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:33906/0 (socket says 172.21.15.174:33906) 2023-12-25T21:14:18.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.109+0000 7f8ec4d22700 1 -- 172.21.15.174:0/3494302881 learned_addr learned my addr 172.21.15.174:0/3494302881 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:18.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.109+0000 7f8ec4d22700 1 -- 172.21.15.174:0/3494302881 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb4005040 con 0x7f8ec00f5410 2023-12-25T21:14:18.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.109+0000 7f8ec4d22700 1 --2- 172.21.15.174:0/3494302881 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec00f5410 0x7f8ec0105630 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f8eb4013510 tx=0x7f8eb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:18.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.109+0000 7f8ebd7fa700 1 -- 172.21.15.174:0/3494302881 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8eb4009320 con 0x7f8ec00f5410 2023-12-25T21:14:18.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.109+0000 7f8ec6f86700 1 -- 172.21.15.174:0/3494302881 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec0105d20 con 0x7f8ec00f5410 2023-12-25T21:14:18.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.109+0000 7f8ebd7fa700 1 -- 172.21.15.174:0/3494302881 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8eb4013f00 con 0x7f8ec00f5410 2023-12-25T21:14:18.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.109+0000 7f8ebd7fa700 1 -- 172.21.15.174:0/3494302881 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8eb4014d90 con 0x7f8ec00f5410 2023-12-25T21:14:18.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.109+0000 7f8ec6f86700 1 -- 172.21.15.174:0/3494302881 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec01061b0 con 0x7f8ec00f5410 2023-12-25T21:14:18.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.110+0000 7f8ebd7fa700 1 -- 172.21.15.174:0/3494302881 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8eb40095f0 con 0x7f8ec00f5410 2023-12-25T21:14:18.112 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.110+0000 7f8ec6f86700 1 -- 172.21.15.174:0/3494302881 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec0045ad0 con 0x7f8ec00f5410 2023-12-25T21:14:18.113 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.110+0000 7f8ebd7fa700 1 --2- 172.21.15.174:0/3494302881 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8eb002ff60 0x7f8eb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:18.113 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.110+0000 7f8ebd7fa700 1 -- 172.21.15.174:0/3494302881 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8eb4041080 con 0x7f8ec00f5410 2023-12-25T21:14:18.113 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.110+0000 7f8ebffff700 1 --2- 172.21.15.174:0/3494302881 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8eb002ff60 0x7f8eb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:18.113 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.111+0000 7f8ebffff700 1 --2- 172.21.15.174:0/3494302881 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8eb002ff60 0x7f8eb0032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f8eac0096a0 tx=0x7f8eac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:18.116 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.115+0000 7f8ebd7fa700 1 -- 172.21.15.174:0/3494302881 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8eb401d7f0 con 0x7f8ec00f5410 2023-12-25T21:14:18.320 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.319+0000 7f8ec6f86700 1 -- 172.21.15.174:0/3494302881 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ec00fae60 con 0x7f8ec00f5410 2023-12-25T21:14:18.320 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.320+0000 7f8ebd7fa700 1 -- 172.21.15.174:0/3494302881 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8eb4010230 con 0x7f8ec00f5410 2023-12-25T21:14:18.321 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:18.321 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:18.322 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.321+0000 7f8ec6f86700 1 -- 172.21.15.174:0/3494302881 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8eb002ff60 msgr2=0x7f8eb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:18.322 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.321+0000 7f8ec6f86700 1 --2- 172.21.15.174:0/3494302881 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8eb002ff60 0x7f8eb0032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f8eac0096a0 tx=0x7f8eac006b40).stop 2023-12-25T21:14:18.322 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.321+0000 7f8ec6f86700 1 -- 172.21.15.174:0/3494302881 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec00f5410 msgr2=0x7f8ec0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:18.322 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.321+0000 7f8ec6f86700 1 --2- 172.21.15.174:0/3494302881 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec00f5410 0x7f8ec0105630 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f8eb4013510 tx=0x7f8eb400b040).stop 2023-12-25T21:14:18.322 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.321+0000 7f8ec6f86700 1 -- 172.21.15.174:0/3494302881 shutdown_connections 2023-12-25T21:14:18.322 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.321+0000 7f8ec6f86700 1 --2- 172.21.15.174:0/3494302881 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f8eb002ff60 0x7f8eb0032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:18.323 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.321+0000 7f8ec6f86700 1 --2- 172.21.15.174:0/3494302881 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec00f5410 0x7f8ec0105630 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:18.323 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.321+0000 7f8ec6f86700 1 -- 172.21.15.174:0/3494302881 >> 172.21.15.174:0/3494302881 conn(0x7f8ec00f0b70 msgr2=0x7f8ec00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:18.323 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.321+0000 7f8ec6f86700 1 -- 172.21.15.174:0/3494302881 shutdown_connections 2023-12-25T21:14:18.323 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:18.321+0000 7f8ec6f86700 1 -- 172.21.15.174:0/3494302881 wait complete. 2023-12-25T21:14:18.323 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:18.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:18 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:16.637304+0000 mgr.smithi061.jxxeke (mgr.14164) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:18.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:18 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:18.321141+0000 mon.smithi061 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.174:0/3494302881' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:19.723 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:19.724 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:20.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:20 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:18.637595+0000 mgr.smithi061.jxxeke (mgr.14164) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:20.834 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.831+0000 7f239b082700 1 -- 172.21.15.174:0/1335501679 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23940f6510 msgr2=0x7f23940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:20.834 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.831+0000 7f239b082700 1 --2- 172.21.15.174:0/1335501679 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23940f6510 0x7f23940f6930 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f2388009fb0 tx=0x7f2388009b30).stop 2023-12-25T21:14:20.834 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.831+0000 7f239b082700 1 -- 172.21.15.174:0/1335501679 shutdown_connections 2023-12-25T21:14:20.834 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.831+0000 7f239b082700 1 --2- 172.21.15.174:0/1335501679 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23940f6510 0x7f23940f6930 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:20.834 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.831+0000 7f239b082700 1 -- 172.21.15.174:0/1335501679 >> 172.21.15.174:0/1335501679 conn(0x7f23940f1c70 msgr2=0x7f23940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:20.834 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.831+0000 7f239b082700 1 -- 172.21.15.174:0/1335501679 shutdown_connections 2023-12-25T21:14:20.834 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.831+0000 7f239b082700 1 -- 172.21.15.174:0/1335501679 wait complete. 2023-12-25T21:14:20.834 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.832+0000 7f239b082700 1 Processor -- start 2023-12-25T21:14:20.834 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.832+0000 7f239b082700 1 -- start start 2023-12-25T21:14:20.834 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.832+0000 7f239b082700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23940f6510 0x7f2394106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:20.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.832+0000 7f239b082700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2394106c20 con 0x7f23940f6510 2023-12-25T21:14:20.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.832+0000 7f2398e1e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23940f6510 0x7f2394106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:20.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.832+0000 7f2398e1e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23940f6510 0x7f2394106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:33928/0 (socket says 172.21.15.174:33928) 2023-12-25T21:14:20.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.832+0000 7f2398e1e700 1 -- 172.21.15.174:0/4073238792 learned_addr learned my addr 172.21.15.174:0/4073238792 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:20.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.833+0000 7f2398e1e700 1 -- 172.21.15.174:0/4073238792 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2388005040 con 0x7f23940f6510 2023-12-25T21:14:20.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.833+0000 7f2398e1e700 1 --2- 172.21.15.174:0/4073238792 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23940f6510 0x7f2394106730 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f2388013510 tx=0x7f238800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:20.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.833+0000 7f23917fa700 1 -- 172.21.15.174:0/4073238792 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2388009320 con 0x7f23940f6510 2023-12-25T21:14:20.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.833+0000 7f23917fa700 1 -- 172.21.15.174:0/4073238792 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2388013f00 con 0x7f23940f6510 2023-12-25T21:14:20.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.833+0000 7f239b082700 1 -- 172.21.15.174:0/4073238792 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2394106e20 con 0x7f23940f6510 2023-12-25T21:14:20.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.833+0000 7f23917fa700 1 -- 172.21.15.174:0/4073238792 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2388014d90 con 0x7f23940f6510 2023-12-25T21:14:20.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.833+0000 7f239b082700 1 -- 172.21.15.174:0/4073238792 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23941072b0 con 0x7f23940f6510 2023-12-25T21:14:20.836 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.834+0000 7f23917fa700 1 -- 172.21.15.174:0/4073238792 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23880095f0 con 0x7f23940f6510 2023-12-25T21:14:20.836 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.834+0000 7f239b082700 1 -- 172.21.15.174:0/4073238792 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2394045ad0 con 0x7f23940f6510 2023-12-25T21:14:20.836 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.834+0000 7f23917fa700 1 --2- 172.21.15.174:0/4073238792 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f238402ff60 0x7f2384032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:20.836 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.834+0000 7f23917fa700 1 -- 172.21.15.174:0/4073238792 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2388040fe0 con 0x7f23940f6510 2023-12-25T21:14:20.836 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.834+0000 7f2393fff700 1 --2- 172.21.15.174:0/4073238792 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f238402ff60 0x7f2384032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:20.836 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.835+0000 7f2393fff700 1 --2- 172.21.15.174:0/4073238792 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f238402ff60 0x7f2384032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f23800096a0 tx=0x7f2380006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:20.839 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:20.839+0000 7f23917fa700 1 -- 172.21.15.174:0/4073238792 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f238801d7f0 con 0x7f23940f6510 2023-12-25T21:14:21.046 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:21.046+0000 7f239b082700 1 -- 172.21.15.174:0/4073238792 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23940fbf60 con 0x7f23940f6510 2023-12-25T21:14:21.047 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:21.046+0000 7f23917fa700 1 -- 172.21.15.174:0/4073238792 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2388010230 con 0x7f23940f6510 2023-12-25T21:14:21.047 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:21.047 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:21.049 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:21.048+0000 7f239b082700 1 -- 172.21.15.174:0/4073238792 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f238402ff60 msgr2=0x7f2384032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:21.049 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:21.048+0000 7f239b082700 1 --2- 172.21.15.174:0/4073238792 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f238402ff60 0x7f2384032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f23800096a0 tx=0x7f2380006b40).stop 2023-12-25T21:14:21.049 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:21.048+0000 7f239b082700 1 -- 172.21.15.174:0/4073238792 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23940f6510 msgr2=0x7f2394106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:21.049 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:21.048+0000 7f239b082700 1 --2- 172.21.15.174:0/4073238792 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23940f6510 0x7f2394106730 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f2388013510 tx=0x7f238800b040).stop 2023-12-25T21:14:21.049 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:21.048+0000 7f239b082700 1 -- 172.21.15.174:0/4073238792 shutdown_connections 2023-12-25T21:14:21.049 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:21.048+0000 7f239b082700 1 --2- 172.21.15.174:0/4073238792 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f238402ff60 0x7f2384032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:21.049 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:21.048+0000 7f239b082700 1 --2- 172.21.15.174:0/4073238792 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23940f6510 0x7f2394106730 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:21.049 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:21.048+0000 7f239b082700 1 -- 172.21.15.174:0/4073238792 >> 172.21.15.174:0/4073238792 conn(0x7f23940f1c70 msgr2=0x7f23940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:21.050 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:21.048+0000 7f239b082700 1 -- 172.21.15.174:0/4073238792 shutdown_connections 2023-12-25T21:14:21.050 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:21.048+0000 7f239b082700 1 -- 172.21.15.174:0/4073238792 wait complete. 2023-12-25T21:14:21.050 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:21.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:21 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:21.047724+0000 mon.smithi061 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.174:0/4073238792' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:22.443 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:22.444 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:22.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:22 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:20.637954+0000 mgr.smithi061.jxxeke (mgr.14164) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:23.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.577+0000 7fd5a6ee8700 1 -- 172.21.15.174:0/2841652960 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a00f6510 msgr2=0x7fd5a00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:23.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.577+0000 7fd5a6ee8700 1 --2- 172.21.15.174:0/2841652960 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a00f6510 0x7fd5a00f6930 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fd594009fb0 tx=0x7fd594009b30).stop 2023-12-25T21:14:23.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.577+0000 7fd5a6ee8700 1 -- 172.21.15.174:0/2841652960 shutdown_connections 2023-12-25T21:14:23.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.577+0000 7fd5a6ee8700 1 --2- 172.21.15.174:0/2841652960 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a00f6510 0x7fd5a00f6930 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:23.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.577+0000 7fd5a6ee8700 1 -- 172.21.15.174:0/2841652960 >> 172.21.15.174:0/2841652960 conn(0x7fd5a00f1c70 msgr2=0x7fd5a00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:23.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.577+0000 7fd5a6ee8700 1 -- 172.21.15.174:0/2841652960 shutdown_connections 2023-12-25T21:14:23.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.577+0000 7fd5a6ee8700 1 -- 172.21.15.174:0/2841652960 wait complete. 2023-12-25T21:14:23.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.578+0000 7fd5a6ee8700 1 Processor -- start 2023-12-25T21:14:23.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.578+0000 7fd5a6ee8700 1 -- start start 2023-12-25T21:14:23.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.578+0000 7fd5a6ee8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a00f6510 0x7fd5a0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:23.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.578+0000 7fd5a6ee8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a0106c20 con 0x7fd5a00f6510 2023-12-25T21:14:23.581 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.578+0000 7fd5a4c84700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a00f6510 0x7fd5a0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:23.581 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.578+0000 7fd5a4c84700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a00f6510 0x7fd5a0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:33938/0 (socket says 172.21.15.174:33938) 2023-12-25T21:14:23.581 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.578+0000 7fd5a4c84700 1 -- 172.21.15.174:0/3975457488 learned_addr learned my addr 172.21.15.174:0/3975457488 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:23.581 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.579+0000 7fd5a4c84700 1 -- 172.21.15.174:0/3975457488 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd594005040 con 0x7fd5a00f6510 2023-12-25T21:14:23.581 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.579+0000 7fd5a4c84700 1 --2- 172.21.15.174:0/3975457488 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a00f6510 0x7fd5a0106730 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fd594014000 tx=0x7fd59400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:23.581 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.579+0000 7fd59d7fa700 1 -- 172.21.15.174:0/3975457488 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5940095f0 con 0x7fd5a00f6510 2023-12-25T21:14:23.582 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.579+0000 7fd5a6ee8700 1 -- 172.21.15.174:0/3975457488 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a0106e20 con 0x7fd5a00f6510 2023-12-25T21:14:23.582 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.579+0000 7fd59d7fa700 1 -- 172.21.15.174:0/3975457488 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd594007e70 con 0x7fd5a00f6510 2023-12-25T21:14:23.582 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.579+0000 7fd59d7fa700 1 -- 172.21.15.174:0/3975457488 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd594014ea0 con 0x7fd5a00f6510 2023-12-25T21:14:23.582 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.579+0000 7fd5a6ee8700 1 -- 172.21.15.174:0/3975457488 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a01072b0 con 0x7fd5a00f6510 2023-12-25T21:14:23.582 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.580+0000 7fd59d7fa700 1 -- 172.21.15.174:0/3975457488 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5940079c0 con 0x7fd5a00f6510 2023-12-25T21:14:23.582 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.580+0000 7fd5a6ee8700 1 -- 172.21.15.174:0/3975457488 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a0045ad0 con 0x7fd5a00f6510 2023-12-25T21:14:23.582 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.580+0000 7fd59d7fa700 1 --2- 172.21.15.174:0/3975457488 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd59002ffb0 0x7fd590032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:23.582 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.580+0000 7fd59d7fa700 1 -- 172.21.15.174:0/3975457488 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd594040920 con 0x7fd5a00f6510 2023-12-25T21:14:23.582 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.580+0000 7fd59ffff700 1 --2- 172.21.15.174:0/3975457488 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd59002ffb0 0x7fd590032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:23.582 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.581+0000 7fd59ffff700 1 --2- 172.21.15.174:0/3975457488 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd59002ffb0 0x7fd590032470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fd58c0096a0 tx=0x7fd58c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:23.586 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.585+0000 7fd59d7fa700 1 -- 172.21.15.174:0/3975457488 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd59400f950 con 0x7fd5a00f6510 2023-12-25T21:14:23.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cephadm 2023-12-25T21:14:22.496244+0000 mgr.smithi061.jxxeke (mgr.14164) 231 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi061 ... 2023-12-25T21:14:23.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Verifying port 9100 ... 2023-12-25T21:14:23.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Non-zero exit code 1 from systemctl start ceph-5532d712-a369-11ee-95a5-87774f69a715@node-exporter.smithi061 2023-12-25T21:14:23.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: systemctl: stderr Job for ceph-5532d712-a369-11ee-95a5-87774f69a715@node-exporter.smithi061.service failed because a timeout was exceeded. 2023-12-25T21:14:23.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: systemctl: stderr See "systemctl status ceph-5532d712-a369-11ee-95a5-87774f69a715@node-exporter.smithi061.service" and "journalctl -xe" for details. 2023-12-25T21:14:23.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Traceback (most recent call last): 2023-12-25T21:14:23.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-25T21:14:23.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: main() 2023-12-25T21:14:23.706 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-25T21:14:23.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: r = ctx.func(ctx) 2023-12-25T21:14:23.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-25T21:14:23.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: return func(ctx) 2023-12-25T21:14:23.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-25T21:14:23.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: ports=daemon_ports) 2023-12-25T21:14:23.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-12-25T21:14:23.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: c, osd_fsid=osd_fsid, ports=ports) 2023-12-25T21:14:23.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-12-25T21:14:23.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-12-25T21:14:23.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-25T21:14:23.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-25T21:14:23.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: RuntimeError: Failed command: systemctl start ceph-5532d712-a369-11ee-95a5-87774f69a715@node-exporter.smithi061 2023-12-25T21:14:23.707 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Traceback (most recent call last): 2023-12-25T21:14:23.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-25T21:14:23.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: yield (conn, connr) 2023-12-25T21:14:23.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-25T21:14:23.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: code, '\n'.join(err))) 2023-12-25T21:14:23.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi061 ... 2023-12-25T21:14:23.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Verifying port 9100 ... 2023-12-25T21:14:23.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Non-zero exit code 1 from systemctl start ceph-5532d712-a369-11ee-95a5-87774f69a715@node-exporter.smithi061 2023-12-25T21:14:23.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: systemctl: stderr Job for ceph-5532d712-a369-11ee-95a5-87774f69a715@node-exporter.smithi061.service failed because a timeout was exceeded. 2023-12-25T21:14:23.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: systemctl: stderr See "systemctl status ceph-5532d712-a369-11ee-95a5-87774f69a715@node-exporter.smithi061.service" and "journalctl -xe" for details. 2023-12-25T21:14:23.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: Traceback (most recent call last): 2023-12-25T21:14:23.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-25T21:14:23.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: main() 2023-12-25T21:14:23.708 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-25T21:14:23.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: r = ctx.func(ctx) 2023-12-25T21:14:23.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-25T21:14:23.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: return func(ctx) 2023-12-25T21:14:23.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-25T21:14:23.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: ports=daemon_ports) 2023-12-25T21:14:23.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-12-25T21:14:23.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: c, osd_fsid=osd_fsid, ports=ports) 2023-12-25T21:14:23.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-12-25T21:14:23.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-12-25T21:14:23.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: File "/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-25T21:14:23.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-25T21:14:23.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: RuntimeError: Failed command: systemctl start ceph-5532d712-a369-11ee-95a5-87774f69a715@node-exporter.smithi061 2023-12-25T21:14:23.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cephadm 2023-12-25T21:14:22.501889+0000 mgr.smithi061.jxxeke (mgr.14164) 232 : cephadm [INF] Deploying daemon prometheus.smithi061 on smithi061 2023-12-25T21:14:23.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:22.638231+0000 mgr.smithi061.jxxeke (mgr.14164) 233 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:23.794 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.793+0000 7fd5a6ee8700 1 -- 172.21.15.174:0/3975457488 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5a00fbf60 con 0x7fd5a00f6510 2023-12-25T21:14:23.795 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.794+0000 7fd59d7fa700 1 -- 172.21.15.174:0/3975457488 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd594010230 con 0x7fd5a00f6510 2023-12-25T21:14:23.795 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:23.795 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:23.796 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.795+0000 7fd5a6ee8700 1 -- 172.21.15.174:0/3975457488 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd59002ffb0 msgr2=0x7fd590032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:23.796 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.795+0000 7fd5a6ee8700 1 --2- 172.21.15.174:0/3975457488 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd59002ffb0 0x7fd590032470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fd58c0096a0 tx=0x7fd58c006b40).stop 2023-12-25T21:14:23.796 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.796+0000 7fd5a6ee8700 1 -- 172.21.15.174:0/3975457488 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a00f6510 msgr2=0x7fd5a0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:23.797 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.796+0000 7fd5a6ee8700 1 --2- 172.21.15.174:0/3975457488 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a00f6510 0x7fd5a0106730 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fd594014000 tx=0x7fd59400b040).stop 2023-12-25T21:14:23.797 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.796+0000 7fd5a6ee8700 1 -- 172.21.15.174:0/3975457488 shutdown_connections 2023-12-25T21:14:23.797 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.796+0000 7fd5a6ee8700 1 --2- 172.21.15.174:0/3975457488 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd59002ffb0 0x7fd590032470 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:23.797 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.796+0000 7fd5a6ee8700 1 --2- 172.21.15.174:0/3975457488 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5a00f6510 0x7fd5a0106730 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:23.797 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.796+0000 7fd5a6ee8700 1 -- 172.21.15.174:0/3975457488 >> 172.21.15.174:0/3975457488 conn(0x7fd5a00f1c70 msgr2=0x7fd5a00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:23.797 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.796+0000 7fd5a6ee8700 1 -- 172.21.15.174:0/3975457488 shutdown_connections 2023-12-25T21:14:23.797 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:23.796+0000 7fd5a6ee8700 1 -- 172.21.15.174:0/3975457488 wait complete. 2023-12-25T21:14:23.797 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:24.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:24 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:23.795527+0000 mon.smithi061 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.174:0/3975457488' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:25.186 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:25.186 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:25.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:25 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:24.638560+0000 mgr.smithi061.jxxeke (mgr.14164) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:26.332 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.329+0000 7f23ac1af700 1 -- 172.21.15.174:0/2836739143 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23a40f5410 msgr2=0x7f23a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:26.332 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.329+0000 7f23ac1af700 1 --2- 172.21.15.174:0/2836739143 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23a40f5410 0x7f23a40f5830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f23a0009fb0 tx=0x7f23a0009b30).stop 2023-12-25T21:14:26.332 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.329+0000 7f23ac1af700 1 -- 172.21.15.174:0/2836739143 shutdown_connections 2023-12-25T21:14:26.332 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.329+0000 7f23ac1af700 1 --2- 172.21.15.174:0/2836739143 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23a40f5410 0x7f23a40f5830 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:26.332 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.329+0000 7f23ac1af700 1 -- 172.21.15.174:0/2836739143 >> 172.21.15.174:0/2836739143 conn(0x7f23a40f0b70 msgr2=0x7f23a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:26.332 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.329+0000 7f23ac1af700 1 -- 172.21.15.174:0/2836739143 shutdown_connections 2023-12-25T21:14:26.332 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.329+0000 7f23ac1af700 1 -- 172.21.15.174:0/2836739143 wait complete. 2023-12-25T21:14:26.332 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.330+0000 7f23ac1af700 1 Processor -- start 2023-12-25T21:14:26.333 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.330+0000 7f23ac1af700 1 -- start start 2023-12-25T21:14:26.333 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.330+0000 7f23ac1af700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23a40f5410 0x7f23a40fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:26.333 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.330+0000 7f23ac1af700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a40fb820 con 0x7f23a40f5410 2023-12-25T21:14:26.333 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.330+0000 7f23a9f4b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23a40f5410 0x7f23a40fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:26.333 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.330+0000 7f23a9f4b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23a40f5410 0x7f23a40fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:33954/0 (socket says 172.21.15.174:33954) 2023-12-25T21:14:26.333 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.330+0000 7f23a9f4b700 1 -- 172.21.15.174:0/2351208581 learned_addr learned my addr 172.21.15.174:0/2351208581 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:26.333 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.331+0000 7f23a9f4b700 1 -- 172.21.15.174:0/2351208581 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a0005040 con 0x7f23a40f5410 2023-12-25T21:14:26.333 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.331+0000 7f23a9f4b700 1 --2- 172.21.15.174:0/2351208581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23a40f5410 0x7f23a40fd3a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f23a0013510 tx=0x7f23a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:26.333 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.331+0000 7f239a7fc700 1 -- 172.21.15.174:0/2351208581 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23a0009320 con 0x7f23a40f5410 2023-12-25T21:14:26.333 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.331+0000 7f23ac1af700 1 -- 172.21.15.174:0/2351208581 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a40fb9c0 con 0x7f23a40f5410 2023-12-25T21:14:26.333 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.331+0000 7f239a7fc700 1 -- 172.21.15.174:0/2351208581 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23a0013f00 con 0x7f23a40f5410 2023-12-25T21:14:26.333 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.331+0000 7f239a7fc700 1 -- 172.21.15.174:0/2351208581 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23a0014d90 con 0x7f23a40f5410 2023-12-25T21:14:26.333 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.331+0000 7f23ac1af700 1 -- 172.21.15.174:0/2351208581 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a40fbe50 con 0x7f23a40f5410 2023-12-25T21:14:26.334 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.332+0000 7f239a7fc700 1 -- 172.21.15.174:0/2351208581 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23a00095f0 con 0x7f23a40f5410 2023-12-25T21:14:26.334 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.332+0000 7f23ac1af700 1 -- 172.21.15.174:0/2351208581 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a4059100 con 0x7f23a40f5410 2023-12-25T21:14:26.335 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.332+0000 7f239a7fc700 1 --2- 172.21.15.174:0/2351208581 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f239002ff60 0x7f2390032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:26.335 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.332+0000 7f239a7fc700 1 -- 172.21.15.174:0/2351208581 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f23a0040fc0 con 0x7f23a40f5410 2023-12-25T21:14:26.335 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.332+0000 7f23a974a700 1 --2- 172.21.15.174:0/2351208581 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f239002ff60 0x7f2390032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:26.335 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.333+0000 7f23a974a700 1 --2- 172.21.15.174:0/2351208581 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f239002ff60 0x7f2390032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f23940096a0 tx=0x7f2394006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:26.337 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.337+0000 7f239a7fc700 1 -- 172.21.15.174:0/2351208581 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23a0014420 con 0x7f23a40f5410 2023-12-25T21:14:26.544 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.543+0000 7f23ac1af700 1 -- 172.21.15.174:0/2351208581 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23a4045ad0 con 0x7f23a40f5410 2023-12-25T21:14:26.545 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.544+0000 7f239a7fc700 1 -- 172.21.15.174:0/2351208581 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f23a0010230 con 0x7f23a40f5410 2023-12-25T21:14:26.545 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:26.545 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:26.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.546+0000 7f23ac1af700 1 -- 172.21.15.174:0/2351208581 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f239002ff60 msgr2=0x7f2390032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:26.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.546+0000 7f23ac1af700 1 --2- 172.21.15.174:0/2351208581 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f239002ff60 0x7f2390032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f23940096a0 tx=0x7f2394006b40).stop 2023-12-25T21:14:26.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.546+0000 7f23ac1af700 1 -- 172.21.15.174:0/2351208581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23a40f5410 msgr2=0x7f23a40fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:26.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.546+0000 7f23ac1af700 1 --2- 172.21.15.174:0/2351208581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23a40f5410 0x7f23a40fd3a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f23a0013510 tx=0x7f23a000b040).stop 2023-12-25T21:14:26.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.546+0000 7f23ac1af700 1 -- 172.21.15.174:0/2351208581 shutdown_connections 2023-12-25T21:14:26.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.546+0000 7f23ac1af700 1 --2- 172.21.15.174:0/2351208581 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f239002ff60 0x7f2390032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:26.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.546+0000 7f23ac1af700 1 --2- 172.21.15.174:0/2351208581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23a40f5410 0x7f23a40fd3a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:26.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.546+0000 7f23ac1af700 1 -- 172.21.15.174:0/2351208581 >> 172.21.15.174:0/2351208581 conn(0x7f23a40f0b70 msgr2=0x7f23a405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:26.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.546+0000 7f23ac1af700 1 -- 172.21.15.174:0/2351208581 shutdown_connections 2023-12-25T21:14:26.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:26.546+0000 7f23ac1af700 1 -- 172.21.15.174:0/2351208581 wait complete. 2023-12-25T21:14:26.548 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:27.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:27 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:26.545505+0000 mon.smithi061 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.174:0/2351208581' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:27.988 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:27.988 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:28.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:28 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:26.638902+0000 mgr.smithi061.jxxeke (mgr.14164) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:29.130 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.129+0000 7f36851f3700 1 -- 172.21.15.174:0/3285065928 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36800f5410 msgr2=0x7f36800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:29.131 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.129+0000 7f36851f3700 1 --2- 172.21.15.174:0/3285065928 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36800f5410 0x7f36800f5830 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f3674009fb0 tx=0x7f3674009b30).stop 2023-12-25T21:14:29.131 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.129+0000 7f36851f3700 1 -- 172.21.15.174:0/3285065928 shutdown_connections 2023-12-25T21:14:29.131 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.129+0000 7f36851f3700 1 --2- 172.21.15.174:0/3285065928 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36800f5410 0x7f36800f5830 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:29.131 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.129+0000 7f36851f3700 1 -- 172.21.15.174:0/3285065928 >> 172.21.15.174:0/3285065928 conn(0x7f36800f0b70 msgr2=0x7f36800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:29.131 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.130+0000 7f36851f3700 1 -- 172.21.15.174:0/3285065928 shutdown_connections 2023-12-25T21:14:29.131 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.130+0000 7f36851f3700 1 -- 172.21.15.174:0/3285065928 wait complete. 2023-12-25T21:14:29.132 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.130+0000 7f36851f3700 1 Processor -- start 2023-12-25T21:14:29.132 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.130+0000 7f36851f3700 1 -- start start 2023-12-25T21:14:29.133 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.130+0000 7f36851f3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3680060200 0x7f3680109580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:29.133 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.130+0000 7f36851f3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3680109a70 con 0x7f3680060200 2023-12-25T21:14:29.133 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.131+0000 7f367ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3680060200 0x7f3680109580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:29.133 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.131+0000 7f367ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3680060200 0x7f3680109580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:36152/0 (socket says 172.21.15.174:36152) 2023-12-25T21:14:29.133 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.131+0000 7f367ed9d700 1 -- 172.21.15.174:0/1961776246 learned_addr learned my addr 172.21.15.174:0/1961776246 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:29.133 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.131+0000 7f367ed9d700 1 -- 172.21.15.174:0/1961776246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3674005040 con 0x7f3680060200 2023-12-25T21:14:29.133 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.131+0000 7f367ed9d700 1 --2- 172.21.15.174:0/1961776246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3680060200 0x7f3680109580 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f3674009f80 tx=0x7f3674009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:29.133 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.131+0000 7f36677fe700 1 -- 172.21.15.174:0/1961776246 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36740136c0 con 0x7f3680060200 2023-12-25T21:14:29.133 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.131+0000 7f36851f3700 1 -- 172.21.15.174:0/1961776246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3680109bb0 con 0x7f3680060200 2023-12-25T21:14:29.133 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.131+0000 7f36677fe700 1 -- 172.21.15.174:0/1961776246 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3674013e70 con 0x7f3680060200 2023-12-25T21:14:29.133 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.131+0000 7f36677fe700 1 -- 172.21.15.174:0/1961776246 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3674014420 con 0x7f3680060200 2023-12-25T21:14:29.133 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.132+0000 7f36851f3700 1 -- 172.21.15.174:0/1961776246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3680109fe0 con 0x7f3680060200 2023-12-25T21:14:29.134 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.132+0000 7f36677fe700 1 -- 172.21.15.174:0/1961776246 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3674025060 con 0x7f3680060200 2023-12-25T21:14:29.134 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.132+0000 7f36851f3700 1 -- 172.21.15.174:0/1961776246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3680045ad0 con 0x7f3680060200 2023-12-25T21:14:29.135 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.132+0000 7f36677fe700 1 --2- 172.21.15.174:0/1961776246 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f366802ff60 0x7f3668032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:29.135 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.133+0000 7f36677fe700 1 -- 172.21.15.174:0/1961776246 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3674040ec0 con 0x7f3680060200 2023-12-25T21:14:29.135 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.133+0000 7f367e59c700 1 --2- 172.21.15.174:0/1961776246 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f366802ff60 0x7f3668032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:29.135 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.133+0000 7f367e59c700 1 --2- 172.21.15.174:0/1961776246 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f366802ff60 0x7f3668032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f36700096a0 tx=0x7f3670006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:29.138 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.138+0000 7f36677fe700 1 -- 172.21.15.174:0/1961776246 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f367401cd10 con 0x7f3680060200 2023-12-25T21:14:29.348 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.346+0000 7f36851f3700 1 -- 172.21.15.174:0/1961776246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f36800fae60 con 0x7f3680060200 2023-12-25T21:14:29.348 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.347+0000 7f36677fe700 1 -- 172.21.15.174:0/1961776246 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f367402d2d0 con 0x7f3680060200 2023-12-25T21:14:29.349 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:29.349 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:29.351 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.349+0000 7f36851f3700 1 -- 172.21.15.174:0/1961776246 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f366802ff60 msgr2=0x7f3668032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:29.351 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.349+0000 7f36851f3700 1 --2- 172.21.15.174:0/1961776246 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f366802ff60 0x7f3668032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f36700096a0 tx=0x7f3670006b40).stop 2023-12-25T21:14:29.351 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.349+0000 7f36851f3700 1 -- 172.21.15.174:0/1961776246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3680060200 msgr2=0x7f3680109580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:29.351 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.349+0000 7f36851f3700 1 --2- 172.21.15.174:0/1961776246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3680060200 0x7f3680109580 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f3674009f80 tx=0x7f3674009b30).stop 2023-12-25T21:14:29.351 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.349+0000 7f36851f3700 1 -- 172.21.15.174:0/1961776246 shutdown_connections 2023-12-25T21:14:29.352 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.349+0000 7f36851f3700 1 --2- 172.21.15.174:0/1961776246 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f366802ff60 0x7f3668032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:29.352 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.349+0000 7f36851f3700 1 --2- 172.21.15.174:0/1961776246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3680060200 0x7f3680109580 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:29.352 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.349+0000 7f36851f3700 1 -- 172.21.15.174:0/1961776246 >> 172.21.15.174:0/1961776246 conn(0x7f36800f0b70 msgr2=0x7f36800f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:29.352 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.349+0000 7f36851f3700 1 -- 172.21.15.174:0/1961776246 shutdown_connections 2023-12-25T21:14:29.352 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:29.349+0000 7f36851f3700 1 -- 172.21.15.174:0/1961776246 wait complete. 2023-12-25T21:14:29.352 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:30.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:30 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:28.639291+0000 mgr.smithi061.jxxeke (mgr.14164) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:30.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:30 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:29.348623+0000 mon.smithi061 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.174:0/1961776246' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:30.737 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:30.737 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:31.956 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.954+0000 7fad0c22a700 1 -- 172.21.15.174:0/92349988 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad04060280 msgr2=0x7fad040f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:31.956 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.954+0000 7fad0c22a700 1 --2- 172.21.15.174:0/92349988 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad04060280 0x7fad040f90f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7facf4009fb0 tx=0x7facf4009b30).stop 2023-12-25T21:14:31.956 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.954+0000 7fad0c22a700 1 -- 172.21.15.174:0/92349988 shutdown_connections 2023-12-25T21:14:31.956 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.954+0000 7fad0c22a700 1 --2- 172.21.15.174:0/92349988 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad04060280 0x7fad040f90f0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:31.956 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.954+0000 7fad0c22a700 1 -- 172.21.15.174:0/92349988 >> 172.21.15.174:0/92349988 conn(0x7fad040f0b70 msgr2=0x7fad040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:31.956 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.954+0000 7fad0c22a700 1 -- 172.21.15.174:0/92349988 shutdown_connections 2023-12-25T21:14:31.956 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.954+0000 7fad0c22a700 1 -- 172.21.15.174:0/92349988 wait complete. 2023-12-25T21:14:31.956 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.955+0000 7fad0c22a700 1 Processor -- start 2023-12-25T21:14:31.956 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.955+0000 7fad0c22a700 1 -- start start 2023-12-25T21:14:31.957 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.955+0000 7fad0c22a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad04060280 0x7fad041066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:31.957 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.955+0000 7fad0c22a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad04106bd0 con 0x7fad04060280 2023-12-25T21:14:31.957 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.955+0000 7fad09fc6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad04060280 0x7fad041066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:31.957 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.955+0000 7fad09fc6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad04060280 0x7fad041066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:36174/0 (socket says 172.21.15.174:36174) 2023-12-25T21:14:31.957 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.955+0000 7fad09fc6700 1 -- 172.21.15.174:0/119900225 learned_addr learned my addr 172.21.15.174:0/119900225 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:31.958 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.955+0000 7fad09fc6700 1 -- 172.21.15.174:0/119900225 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf4005040 con 0x7fad04060280 2023-12-25T21:14:31.958 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.956+0000 7fad09fc6700 1 --2- 172.21.15.174:0/119900225 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad04060280 0x7fad041066e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7facf4013510 tx=0x7facf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:31.959 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.956+0000 7facfa7fc700 1 -- 172.21.15.174:0/119900225 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facf4009320 con 0x7fad04060280 2023-12-25T21:14:31.959 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.956+0000 7fad0c22a700 1 -- 172.21.15.174:0/119900225 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad04106dd0 con 0x7fad04060280 2023-12-25T21:14:31.959 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.956+0000 7facfa7fc700 1 -- 172.21.15.174:0/119900225 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7facf4013f00 con 0x7fad04060280 2023-12-25T21:14:31.959 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.956+0000 7facfa7fc700 1 -- 172.21.15.174:0/119900225 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facf4014dc0 con 0x7fad04060280 2023-12-25T21:14:31.959 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.956+0000 7fad0c22a700 1 -- 172.21.15.174:0/119900225 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad04107260 con 0x7fad04060280 2023-12-25T21:14:31.959 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.957+0000 7facfa7fc700 1 -- 172.21.15.174:0/119900225 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7facf40095f0 con 0x7fad04060280 2023-12-25T21:14:31.959 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.957+0000 7fad0c22a700 1 -- 172.21.15.174:0/119900225 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad04045ad0 con 0x7fad04060280 2023-12-25T21:14:31.959 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.957+0000 7facfa7fc700 1 --2- 172.21.15.174:0/119900225 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7facf002ff60 0x7facf0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:31.959 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.957+0000 7facfa7fc700 1 -- 172.21.15.174:0/119900225 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7facf4041000 con 0x7fad04060280 2023-12-25T21:14:31.959 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.957+0000 7fad097c5700 1 --2- 172.21.15.174:0/119900225 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7facf002ff60 0x7facf0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:31.959 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.958+0000 7fad097c5700 1 --2- 172.21.15.174:0/119900225 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7facf002ff60 0x7facf0032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fad000096a0 tx=0x7fad00006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:31.962 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:31.962+0000 7facfa7fc700 1 -- 172.21.15.174:0/119900225 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7facf401c790 con 0x7fad04060280 2023-12-25T21:14:32.165 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:32.165+0000 7fad0c22a700 1 -- 172.21.15.174:0/119900225 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad04108110 con 0x7fad04060280 2023-12-25T21:14:32.166 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:32.165+0000 7facfa7fc700 1 -- 172.21.15.174:0/119900225 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7facf4010230 con 0x7fad04060280 2023-12-25T21:14:32.166 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:32.166 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:32.168 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:32.167+0000 7fad0c22a700 1 -- 172.21.15.174:0/119900225 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7facf002ff60 msgr2=0x7facf0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:32.168 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:32.167+0000 7fad0c22a700 1 --2- 172.21.15.174:0/119900225 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7facf002ff60 0x7facf0032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fad000096a0 tx=0x7fad00006b40).stop 2023-12-25T21:14:32.168 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:32.167+0000 7fad0c22a700 1 -- 172.21.15.174:0/119900225 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad04060280 msgr2=0x7fad041066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:32.168 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:32.167+0000 7fad0c22a700 1 --2- 172.21.15.174:0/119900225 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad04060280 0x7fad041066e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7facf4013510 tx=0x7facf400b040).stop 2023-12-25T21:14:32.168 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:32.167+0000 7fad0c22a700 1 -- 172.21.15.174:0/119900225 shutdown_connections 2023-12-25T21:14:32.168 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:32.167+0000 7fad0c22a700 1 --2- 172.21.15.174:0/119900225 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7facf002ff60 0x7facf0032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:32.168 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:32.167+0000 7fad0c22a700 1 --2- 172.21.15.174:0/119900225 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad04060280 0x7fad041066e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:32.168 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:32.167+0000 7fad0c22a700 1 -- 172.21.15.174:0/119900225 >> 172.21.15.174:0/119900225 conn(0x7fad040f0b70 msgr2=0x7fad040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:32.168 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:32.167+0000 7fad0c22a700 1 -- 172.21.15.174:0/119900225 shutdown_connections 2023-12-25T21:14:32.169 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:32.167+0000 7fad0c22a700 1 -- 172.21.15.174:0/119900225 wait complete. 2023-12-25T21:14:32.169 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:32.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:32 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:30.639652+0000 mgr.smithi061.jxxeke (mgr.14164) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:32.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:32 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:32.166870+0000 mon.smithi061 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.174:0/119900225' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:33.551 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:33.552 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:34.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:34 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:32.639976+0000 mgr.smithi061.jxxeke (mgr.14164) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:34.719 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.718+0000 7fbc61bfd700 1 -- 172.21.15.174:0/1271619469 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc5c0f4740 msgr2=0x7fbc5c0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:34.719 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.718+0000 7fbc61bfd700 1 --2- 172.21.15.174:0/1271619469 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc5c0f4740 0x7fbc5c0f4b60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fbc50009fb0 tx=0x7fbc50009b30).stop 2023-12-25T21:14:34.719 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.718+0000 7fbc61bfd700 1 -- 172.21.15.174:0/1271619469 shutdown_connections 2023-12-25T21:14:34.719 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.718+0000 7fbc61bfd700 1 --2- 172.21.15.174:0/1271619469 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc5c0f4740 0x7fbc5c0f4b60 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:34.719 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.718+0000 7fbc61bfd700 1 -- 172.21.15.174:0/1271619469 >> 172.21.15.174:0/1271619469 conn(0x7fbc5c0f0260 msgr2=0x7fbc5c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:34.719 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.718+0000 7fbc61bfd700 1 -- 172.21.15.174:0/1271619469 shutdown_connections 2023-12-25T21:14:34.719 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.718+0000 7fbc61bfd700 1 -- 172.21.15.174:0/1271619469 wait complete. 2023-12-25T21:14:34.720 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.718+0000 7fbc61bfd700 1 Processor -- start 2023-12-25T21:14:34.720 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.719+0000 7fbc61bfd700 1 -- start start 2023-12-25T21:14:34.721 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.719+0000 7fbc61bfd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc5c0f4740 0x7fbc5c0f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:34.721 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.719+0000 7fbc61bfd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c0f5770 con 0x7fbc5c0f4740 2023-12-25T21:14:34.721 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.719+0000 7fbc5b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc5c0f4740 0x7fbc5c0f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:34.721 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.719+0000 7fbc5b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc5c0f4740 0x7fbc5c0f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:36206/0 (socket says 172.21.15.174:36206) 2023-12-25T21:14:34.721 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.719+0000 7fbc5b7fe700 1 -- 172.21.15.174:0/3780329026 learned_addr learned my addr 172.21.15.174:0/3780329026 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:34.721 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.719+0000 7fbc5b7fe700 1 -- 172.21.15.174:0/3780329026 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc50005040 con 0x7fbc5c0f4740 2023-12-25T21:14:34.721 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.719+0000 7fbc5b7fe700 1 --2- 172.21.15.174:0/3780329026 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc5c0f4740 0x7fbc5c0f72f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fbc50013510 tx=0x7fbc5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:34.721 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.720+0000 7fbc43fff700 1 -- 172.21.15.174:0/3780329026 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc50009320 con 0x7fbc5c0f4740 2023-12-25T21:14:34.721 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.720+0000 7fbc43fff700 1 -- 172.21.15.174:0/3780329026 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc50013f00 con 0x7fbc5c0f4740 2023-12-25T21:14:34.722 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.720+0000 7fbc61bfd700 1 -- 172.21.15.174:0/3780329026 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc5c0f5910 con 0x7fbc5c0f4740 2023-12-25T21:14:34.722 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.720+0000 7fbc43fff700 1 -- 172.21.15.174:0/3780329026 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc50014d90 con 0x7fbc5c0f4740 2023-12-25T21:14:34.722 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.720+0000 7fbc61bfd700 1 -- 172.21.15.174:0/3780329026 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc5c0f5da0 con 0x7fbc5c0f4740 2023-12-25T21:14:34.722 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.720+0000 7fbc43fff700 1 -- 172.21.15.174:0/3780329026 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc500263a0 con 0x7fbc5c0f4740 2023-12-25T21:14:34.722 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.721+0000 7fbc43fff700 1 --2- 172.21.15.174:0/3780329026 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbc4402ff60 0x7fbc44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:34.723 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.721+0000 7fbc43fff700 1 -- 172.21.15.174:0/3780329026 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc500410c0 con 0x7fbc5c0f4740 2023-12-25T21:14:34.723 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.721+0000 7fbc61bfd700 1 -- 172.21.15.174:0/3780329026 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc5c059100 con 0x7fbc5c0f4740 2023-12-25T21:14:34.723 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.721+0000 7fbc5affd700 1 --2- 172.21.15.174:0/3780329026 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbc4402ff60 0x7fbc44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:34.723 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.721+0000 7fbc5affd700 1 --2- 172.21.15.174:0/3780329026 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbc4402ff60 0x7fbc44032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fbc4c0096a0 tx=0x7fbc4c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:34.726 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.725+0000 7fbc43fff700 1 -- 172.21.15.174:0/3780329026 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc50012920 con 0x7fbc5c0f4740 2023-12-25T21:14:34.933 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.932+0000 7fbc61bfd700 1 -- 172.21.15.174:0/3780329026 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc5c0f64a0 con 0x7fbc5c0f4740 2023-12-25T21:14:34.933 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.933+0000 7fbc43fff700 1 -- 172.21.15.174:0/3780329026 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbc5002b020 con 0x7fbc5c0f4740 2023-12-25T21:14:34.934 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:34.934 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:34.935 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.934+0000 7fbc61bfd700 1 -- 172.21.15.174:0/3780329026 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbc4402ff60 msgr2=0x7fbc44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:34.935 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.934+0000 7fbc61bfd700 1 --2- 172.21.15.174:0/3780329026 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbc4402ff60 0x7fbc44032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fbc4c0096a0 tx=0x7fbc4c006b40).stop 2023-12-25T21:14:34.935 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.934+0000 7fbc61bfd700 1 -- 172.21.15.174:0/3780329026 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc5c0f4740 msgr2=0x7fbc5c0f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:34.936 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.934+0000 7fbc61bfd700 1 --2- 172.21.15.174:0/3780329026 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc5c0f4740 0x7fbc5c0f72f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fbc50013510 tx=0x7fbc5000b040).stop 2023-12-25T21:14:34.936 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.934+0000 7fbc61bfd700 1 -- 172.21.15.174:0/3780329026 shutdown_connections 2023-12-25T21:14:34.936 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.934+0000 7fbc61bfd700 1 --2- 172.21.15.174:0/3780329026 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbc4402ff60 0x7fbc44032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:34.936 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.934+0000 7fbc61bfd700 1 --2- 172.21.15.174:0/3780329026 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc5c0f4740 0x7fbc5c0f72f0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:34.936 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.934+0000 7fbc61bfd700 1 -- 172.21.15.174:0/3780329026 >> 172.21.15.174:0/3780329026 conn(0x7fbc5c0f0260 msgr2=0x7fbc5c0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:34.936 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.935+0000 7fbc61bfd700 1 -- 172.21.15.174:0/3780329026 shutdown_connections 2023-12-25T21:14:34.936 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:34.935+0000 7fbc61bfd700 1 -- 172.21.15.174:0/3780329026 wait complete. 2023-12-25T21:14:34.936 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:35.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:35 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:34.934120+0000 mon.smithi061 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.174:0/3780329026' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:36.329 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:36.329 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:36.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:36 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:34.640305+0000 mgr.smithi061.jxxeke (mgr.14164) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:37.590 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.589+0000 7ff7b8b91700 1 -- 172.21.15.174:0/1045746881 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7b4060180 msgr2=0x7ff7b40f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:37.590 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.589+0000 7ff7b8b91700 1 --2- 172.21.15.174:0/1045746881 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7b4060180 0x7ff7b40f8f80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7ff7a4009fb0 tx=0x7ff7a4009b30).stop 2023-12-25T21:14:37.590 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.589+0000 7ff7b8b91700 1 -- 172.21.15.174:0/1045746881 shutdown_connections 2023-12-25T21:14:37.591 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.589+0000 7ff7b8b91700 1 --2- 172.21.15.174:0/1045746881 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7b4060180 0x7ff7b40f8f80 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:37.591 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.589+0000 7ff7b8b91700 1 -- 172.21.15.174:0/1045746881 >> 172.21.15.174:0/1045746881 conn(0x7ff7b40f09a0 msgr2=0x7ff7b40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:37.591 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.589+0000 7ff7b8b91700 1 -- 172.21.15.174:0/1045746881 shutdown_connections 2023-12-25T21:14:37.591 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.589+0000 7ff7b8b91700 1 -- 172.21.15.174:0/1045746881 wait complete. 2023-12-25T21:14:37.591 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.590+0000 7ff7b8b91700 1 Processor -- start 2023-12-25T21:14:37.593 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.590+0000 7ff7b8b91700 1 -- start start 2023-12-25T21:14:37.593 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.590+0000 7ff7b8b91700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7b4060180 0x7ff7b4106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:37.593 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.590+0000 7ff7b8b91700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b41069f0 con 0x7ff7b4060180 2023-12-25T21:14:37.593 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.590+0000 7ff7b37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7b4060180 0x7ff7b4106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:37.593 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.590+0000 7ff7b37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7b4060180 0x7ff7b4106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:47106/0 (socket says 172.21.15.174:47106) 2023-12-25T21:14:37.593 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.590+0000 7ff7b37fe700 1 -- 172.21.15.174:0/1376102212 learned_addr learned my addr 172.21.15.174:0/1376102212 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:37.593 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.591+0000 7ff7b37fe700 1 -- 172.21.15.174:0/1376102212 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a4005040 con 0x7ff7b4060180 2023-12-25T21:14:37.593 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.591+0000 7ff7b37fe700 1 --2- 172.21.15.174:0/1376102212 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7b4060180 0x7ff7b4106500 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7ff7a4013510 tx=0x7ff7a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:37.593 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.591+0000 7ff79bfff700 1 -- 172.21.15.174:0/1376102212 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7a4009320 con 0x7ff7b4060180 2023-12-25T21:14:37.593 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.591+0000 7ff7b8b91700 1 -- 172.21.15.174:0/1376102212 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b4106bf0 con 0x7ff7b4060180 2023-12-25T21:14:37.593 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.591+0000 7ff79bfff700 1 -- 172.21.15.174:0/1376102212 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7a4013f00 con 0x7ff7b4060180 2023-12-25T21:14:37.593 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.591+0000 7ff79bfff700 1 -- 172.21.15.174:0/1376102212 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7a4014d90 con 0x7ff7b4060180 2023-12-25T21:14:37.594 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.591+0000 7ff7b8b91700 1 -- 172.21.15.174:0/1376102212 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b4107080 con 0x7ff7b4060180 2023-12-25T21:14:37.594 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.592+0000 7ff79bfff700 1 -- 172.21.15.174:0/1376102212 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7a40095f0 con 0x7ff7b4060180 2023-12-25T21:14:37.594 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.592+0000 7ff7b8b91700 1 -- 172.21.15.174:0/1376102212 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7b4045ad0 con 0x7ff7b4060180 2023-12-25T21:14:37.594 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.592+0000 7ff79bfff700 1 --2- 172.21.15.174:0/1376102212 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff79c02ff60 0x7ff79c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:37.594 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.592+0000 7ff79bfff700 1 -- 172.21.15.174:0/1376102212 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff7a40410c0 con 0x7ff7b4060180 2023-12-25T21:14:37.594 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.592+0000 7ff7b2ffd700 1 --2- 172.21.15.174:0/1376102212 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff79c02ff60 0x7ff79c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:37.594 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.592+0000 7ff7b2ffd700 1 --2- 172.21.15.174:0/1376102212 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff79c02ff60 0x7ff79c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7ff7a80096a0 tx=0x7ff7a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:37.597 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.597+0000 7ff79bfff700 1 -- 172.21.15.174:0/1376102212 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff7a4014420 con 0x7ff7b4060180 2023-12-25T21:14:37.802 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.802+0000 7ff7b8b91700 1 -- 172.21.15.174:0/1376102212 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7b4107f30 con 0x7ff7b4060180 2023-12-25T21:14:37.803 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.802+0000 7ff79bfff700 1 -- 172.21.15.174:0/1376102212 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff7a4010230 con 0x7ff7b4060180 2023-12-25T21:14:37.803 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:37.803 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:37.805 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.804+0000 7ff7b8b91700 1 -- 172.21.15.174:0/1376102212 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff79c02ff60 msgr2=0x7ff79c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:37.805 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.804+0000 7ff7b8b91700 1 --2- 172.21.15.174:0/1376102212 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff79c02ff60 0x7ff79c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7ff7a80096a0 tx=0x7ff7a8006b40).stop 2023-12-25T21:14:37.805 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.804+0000 7ff7b8b91700 1 -- 172.21.15.174:0/1376102212 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7b4060180 msgr2=0x7ff7b4106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:37.805 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.804+0000 7ff7b8b91700 1 --2- 172.21.15.174:0/1376102212 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7b4060180 0x7ff7b4106500 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7ff7a4013510 tx=0x7ff7a400b040).stop 2023-12-25T21:14:37.805 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.804+0000 7ff7b8b91700 1 -- 172.21.15.174:0/1376102212 shutdown_connections 2023-12-25T21:14:37.805 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.804+0000 7ff7b8b91700 1 --2- 172.21.15.174:0/1376102212 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7ff79c02ff60 0x7ff79c032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:37.805 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.804+0000 7ff7b8b91700 1 --2- 172.21.15.174:0/1376102212 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7b4060180 0x7ff7b4106500 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:37.805 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.804+0000 7ff7b8b91700 1 -- 172.21.15.174:0/1376102212 >> 172.21.15.174:0/1376102212 conn(0x7ff7b40f09a0 msgr2=0x7ff7b40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:37.806 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.804+0000 7ff7b8b91700 1 -- 172.21.15.174:0/1376102212 shutdown_connections 2023-12-25T21:14:37.806 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:37.804+0000 7ff7b8b91700 1 -- 172.21.15.174:0/1376102212 wait complete. 2023-12-25T21:14:37.806 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:38.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:38 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:36.640695+0000 mgr.smithi061.jxxeke (mgr.14164) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:38.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:38 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:37.803814+0000 mon.smithi061 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.174:0/1376102212' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:39.202 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:39.203 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:40.419 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.417+0000 7f42679c7700 1 -- 172.21.15.174:0/3857336339 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f42600601c0 msgr2=0x7f42600f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:40.419 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.417+0000 7f42679c7700 1 --2- 172.21.15.174:0/3857336339 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f42600601c0 0x7f42600f91b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f425c009fb0 tx=0x7f425c009b30).stop 2023-12-25T21:14:40.419 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.417+0000 7f42679c7700 1 -- 172.21.15.174:0/3857336339 shutdown_connections 2023-12-25T21:14:40.419 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.417+0000 7f42679c7700 1 --2- 172.21.15.174:0/3857336339 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f42600601c0 0x7f42600f91b0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:40.419 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.417+0000 7f42679c7700 1 -- 172.21.15.174:0/3857336339 >> 172.21.15.174:0/3857336339 conn(0x7f42600f0b70 msgr2=0x7f42600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:40.419 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.417+0000 7f42679c7700 1 -- 172.21.15.174:0/3857336339 shutdown_connections 2023-12-25T21:14:40.419 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.417+0000 7f42679c7700 1 -- 172.21.15.174:0/3857336339 wait complete. 2023-12-25T21:14:40.419 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.417+0000 7f42679c7700 1 Processor -- start 2023-12-25T21:14:40.419 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.417+0000 7f42679c7700 1 -- start start 2023-12-25T21:14:40.419 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.418+0000 7f42679c7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f42600601c0 0x7f42601088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:40.420 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.418+0000 7f42679c7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4260108da0 con 0x7f42600601c0 2023-12-25T21:14:40.420 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.418+0000 7f4265763700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f42600601c0 0x7f42601088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:40.420 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.418+0000 7f4265763700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f42600601c0 0x7f42601088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:47132/0 (socket says 172.21.15.174:47132) 2023-12-25T21:14:40.420 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.418+0000 7f4265763700 1 -- 172.21.15.174:0/429201105 learned_addr learned my addr 172.21.15.174:0/429201105 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:40.420 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.418+0000 7f4265763700 1 -- 172.21.15.174:0/429201105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425c005040 con 0x7f42600601c0 2023-12-25T21:14:40.420 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.418+0000 7f4265763700 1 --2- 172.21.15.174:0/429201105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f42600601c0 0x7f42601088b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f425c013510 tx=0x7f425c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:40.421 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.418+0000 7f4255ffb700 1 -- 172.21.15.174:0/429201105 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f425c009320 con 0x7f42600601c0 2023-12-25T21:14:40.421 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.418+0000 7f42679c7700 1 -- 172.21.15.174:0/429201105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4260108fa0 con 0x7f42600601c0 2023-12-25T21:14:40.421 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.418+0000 7f4255ffb700 1 -- 172.21.15.174:0/429201105 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f425c013f00 con 0x7f42600601c0 2023-12-25T21:14:40.421 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.418+0000 7f4255ffb700 1 -- 172.21.15.174:0/429201105 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f425c014dc0 con 0x7f42600601c0 2023-12-25T21:14:40.421 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.419+0000 7f42679c7700 1 -- 172.21.15.174:0/429201105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4260109430 con 0x7f42600601c0 2023-12-25T21:14:40.422 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.419+0000 7f4255ffb700 1 -- 172.21.15.174:0/429201105 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f425c01d3d0 con 0x7f42600601c0 2023-12-25T21:14:40.422 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.419+0000 7f4255ffb700 1 --2- 172.21.15.174:0/429201105 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f424c02ff60 0x7f424c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:40.422 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.419+0000 7f4255ffb700 1 -- 172.21.15.174:0/429201105 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f425c0410c0 con 0x7f42600601c0 2023-12-25T21:14:40.422 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.420+0000 7f42679c7700 1 -- 172.21.15.174:0/429201105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4260045ad0 con 0x7f42600601c0 2023-12-25T21:14:40.422 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.420+0000 7f4264f62700 1 --2- 172.21.15.174:0/429201105 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f424c02ff60 0x7f424c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:40.422 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.420+0000 7f4264f62700 1 --2- 172.21.15.174:0/429201105 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f424c02ff60 0x7f424c032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f42500096a0 tx=0x7f4250006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:40.425 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.424+0000 7f4255ffb700 1 -- 172.21.15.174:0/429201105 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f425c012920 con 0x7f42600601c0 2023-12-25T21:14:40.631 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.631+0000 7f42679c7700 1 -- 172.21.15.174:0/429201105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42600fbf50 con 0x7f42600601c0 2023-12-25T21:14:40.632 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.631+0000 7f4255ffb700 1 -- 172.21.15.174:0/429201105 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f425c012330 con 0x7f42600601c0 2023-12-25T21:14:40.632 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:40.632 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.633+0000 7f42679c7700 1 -- 172.21.15.174:0/429201105 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f424c02ff60 msgr2=0x7f424c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.633+0000 7f42679c7700 1 --2- 172.21.15.174:0/429201105 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f424c02ff60 0x7f424c032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f42500096a0 tx=0x7f4250006b40).stop 2023-12-25T21:14:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.633+0000 7f42679c7700 1 -- 172.21.15.174:0/429201105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f42600601c0 msgr2=0x7f42601088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.633+0000 7f42679c7700 1 --2- 172.21.15.174:0/429201105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f42600601c0 0x7f42601088b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f425c013510 tx=0x7f425c00b040).stop 2023-12-25T21:14:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.633+0000 7f42679c7700 1 -- 172.21.15.174:0/429201105 shutdown_connections 2023-12-25T21:14:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.633+0000 7f42679c7700 1 --2- 172.21.15.174:0/429201105 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f424c02ff60 0x7f424c032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.633+0000 7f42679c7700 1 --2- 172.21.15.174:0/429201105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f42600601c0 0x7f42601088b0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.633+0000 7f42679c7700 1 -- 172.21.15.174:0/429201105 >> 172.21.15.174:0/429201105 conn(0x7f42600f0b70 msgr2=0x7f42600f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.633+0000 7f42679c7700 1 -- 172.21.15.174:0/429201105 shutdown_connections 2023-12-25T21:14:40.634 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:40.633+0000 7f42679c7700 1 -- 172.21.15.174:0/429201105 wait complete. 2023-12-25T21:14:40.635 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:40.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:40 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:38.641081+0000 mgr.smithi061.jxxeke (mgr.14164) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:41.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:41 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:40.632756+0000 mon.smithi061 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.174:0/429201105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:42.011 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:42.011 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:42.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:42 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:40.641362+0000 mgr.smithi061.jxxeke (mgr.14164) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:43.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.209+0000 7fe3f649e700 1 -- 172.21.15.174:0/1550909548 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3f00f6510 msgr2=0x7fe3f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:43.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.209+0000 7fe3f649e700 1 --2- 172.21.15.174:0/1550909548 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3f00f6510 0x7fe3f00f6930 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fe3e0009fb0 tx=0x7fe3e0009b30).stop 2023-12-25T21:14:43.210 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.209+0000 7fe3f649e700 1 -- 172.21.15.174:0/1550909548 shutdown_connections 2023-12-25T21:14:43.211 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.209+0000 7fe3f649e700 1 --2- 172.21.15.174:0/1550909548 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3f00f6510 0x7fe3f00f6930 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:43.211 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.209+0000 7fe3f649e700 1 -- 172.21.15.174:0/1550909548 >> 172.21.15.174:0/1550909548 conn(0x7fe3f00f1c70 msgr2=0x7fe3f00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:43.211 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.209+0000 7fe3f649e700 1 -- 172.21.15.174:0/1550909548 shutdown_connections 2023-12-25T21:14:43.211 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.209+0000 7fe3f649e700 1 -- 172.21.15.174:0/1550909548 wait complete. 2023-12-25T21:14:43.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.210+0000 7fe3f649e700 1 Processor -- start 2023-12-25T21:14:43.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.210+0000 7fe3f649e700 1 -- start start 2023-12-25T21:14:43.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.210+0000 7fe3f649e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3f00f6510 0x7fe3f01088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:43.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.210+0000 7fe3f649e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f0108de0 con 0x7fe3f00f6510 2023-12-25T21:14:43.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.211+0000 7fe3effff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3f00f6510 0x7fe3f01088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:43.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.211+0000 7fe3effff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3f00f6510 0x7fe3f01088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:47144/0 (socket says 172.21.15.174:47144) 2023-12-25T21:14:43.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.211+0000 7fe3effff700 1 -- 172.21.15.174:0/907647055 learned_addr learned my addr 172.21.15.174:0/907647055 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:43.212 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.211+0000 7fe3effff700 1 -- 172.21.15.174:0/907647055 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e0005040 con 0x7fe3f00f6510 2023-12-25T21:14:43.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.211+0000 7fe3effff700 1 --2- 172.21.15.174:0/907647055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3f00f6510 0x7fe3f01088f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fe3e0013510 tx=0x7fe3e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:43.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.211+0000 7fe3ecff9700 1 -- 172.21.15.174:0/907647055 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3e0009320 con 0x7fe3f00f6510 2023-12-25T21:14:43.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.211+0000 7fe3ecff9700 1 -- 172.21.15.174:0/907647055 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe3e0013f00 con 0x7fe3f00f6510 2023-12-25T21:14:43.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.211+0000 7fe3f649e700 1 -- 172.21.15.174:0/907647055 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f0108fe0 con 0x7fe3f00f6510 2023-12-25T21:14:43.213 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.211+0000 7fe3ecff9700 1 -- 172.21.15.174:0/907647055 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3e0014d90 con 0x7fe3f00f6510 2023-12-25T21:14:43.214 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.212+0000 7fe3f649e700 1 -- 172.21.15.174:0/907647055 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f0109470 con 0x7fe3f00f6510 2023-12-25T21:14:43.214 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.212+0000 7fe3f649e700 1 -- 172.21.15.174:0/907647055 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f0045ad0 con 0x7fe3f00f6510 2023-12-25T21:14:43.214 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.212+0000 7fe3ecff9700 1 -- 172.21.15.174:0/907647055 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3e0014420 con 0x7fe3f00f6510 2023-12-25T21:14:43.214 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.213+0000 7fe3ecff9700 1 --2- 172.21.15.174:0/907647055 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe3dc02ff60 0x7fe3dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:43.214 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.213+0000 7fe3ecff9700 1 -- 172.21.15.174:0/907647055 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe3e0041000 con 0x7fe3f00f6510 2023-12-25T21:14:43.214 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.213+0000 7fe3ef7fe700 1 --2- 172.21.15.174:0/907647055 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe3dc02ff60 0x7fe3dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:43.215 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.214+0000 7fe3ef7fe700 1 --2- 172.21.15.174:0/907647055 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe3dc02ff60 0x7fe3dc032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fe3d80096a0 tx=0x7fe3d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:43.218 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.218+0000 7fe3ecff9700 1 -- 172.21.15.174:0/907647055 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3e00095f0 con 0x7fe3f00f6510 2023-12-25T21:14:43.422 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.421+0000 7fe3f649e700 1 -- 172.21.15.174:0/907647055 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe3f010a320 con 0x7fe3f00f6510 2023-12-25T21:14:43.422 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.422+0000 7fe3ecff9700 1 -- 172.21.15.174:0/907647055 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe3e001db70 con 0x7fe3f00f6510 2023-12-25T21:14:43.423 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:43.423 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:43.424 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.423+0000 7fe3f649e700 1 -- 172.21.15.174:0/907647055 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe3dc02ff60 msgr2=0x7fe3dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:43.424 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.423+0000 7fe3f649e700 1 --2- 172.21.15.174:0/907647055 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe3dc02ff60 0x7fe3dc032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fe3d80096a0 tx=0x7fe3d8006b40).stop 2023-12-25T21:14:43.424 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.423+0000 7fe3f649e700 1 -- 172.21.15.174:0/907647055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3f00f6510 msgr2=0x7fe3f01088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:43.424 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.423+0000 7fe3f649e700 1 --2- 172.21.15.174:0/907647055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3f00f6510 0x7fe3f01088f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fe3e0013510 tx=0x7fe3e000b040).stop 2023-12-25T21:14:43.425 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.423+0000 7fe3f649e700 1 -- 172.21.15.174:0/907647055 shutdown_connections 2023-12-25T21:14:43.425 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.423+0000 7fe3f649e700 1 --2- 172.21.15.174:0/907647055 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe3dc02ff60 0x7fe3dc032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:43.425 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.423+0000 7fe3f649e700 1 --2- 172.21.15.174:0/907647055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3f00f6510 0x7fe3f01088f0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:43.425 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.423+0000 7fe3f649e700 1 -- 172.21.15.174:0/907647055 >> 172.21.15.174:0/907647055 conn(0x7fe3f00f1c70 msgr2=0x7fe3f00f26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:43.425 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.423+0000 7fe3f649e700 1 -- 172.21.15.174:0/907647055 shutdown_connections 2023-12-25T21:14:43.425 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:43.423+0000 7fe3f649e700 1 -- 172.21.15.174:0/907647055 wait complete. 2023-12-25T21:14:43.425 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:44.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:44 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:42.641655+0000 mgr.smithi061.jxxeke (mgr.14164) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:44.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:44 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:43.423174+0000 mon.smithi061 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.174:0/907647055' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:44.813 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:44.813 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:46.062 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.059+0000 7fa81b31d700 1 -- 172.21.15.174:0/66455299 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8140f5410 msgr2=0x7fa8140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:46.062 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.059+0000 7fa81b31d700 1 --2- 172.21.15.174:0/66455299 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8140f5410 0x7fa8140f5830 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fa808009fb0 tx=0x7fa808009b30).stop 2023-12-25T21:14:46.062 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.059+0000 7fa81b31d700 1 -- 172.21.15.174:0/66455299 shutdown_connections 2023-12-25T21:14:46.062 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.059+0000 7fa81b31d700 1 --2- 172.21.15.174:0/66455299 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8140f5410 0x7fa8140f5830 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:46.062 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.059+0000 7fa81b31d700 1 -- 172.21.15.174:0/66455299 >> 172.21.15.174:0/66455299 conn(0x7fa8140f0b70 msgr2=0x7fa8140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:46.062 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.059+0000 7fa81b31d700 1 -- 172.21.15.174:0/66455299 shutdown_connections 2023-12-25T21:14:46.062 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.059+0000 7fa81b31d700 1 -- 172.21.15.174:0/66455299 wait complete. 2023-12-25T21:14:46.062 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.060+0000 7fa81b31d700 1 Processor -- start 2023-12-25T21:14:46.062 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.060+0000 7fa81b31d700 1 -- start start 2023-12-25T21:14:46.062 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.060+0000 7fa81b31d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8140f5410 0x7fa814106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:46.063 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.060+0000 7fa81b31d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa814106c00 con 0x7fa8140f5410 2023-12-25T21:14:46.063 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.060+0000 7fa8190b9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8140f5410 0x7fa814106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:46.063 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.060+0000 7fa8190b9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8140f5410 0x7fa814106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:47166/0 (socket says 172.21.15.174:47166) 2023-12-25T21:14:46.063 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.060+0000 7fa8190b9700 1 -- 172.21.15.174:0/1791936933 learned_addr learned my addr 172.21.15.174:0/1791936933 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:46.063 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.060+0000 7fa8190b9700 1 -- 172.21.15.174:0/1791936933 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa808005040 con 0x7fa8140f5410 2023-12-25T21:14:46.063 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.061+0000 7fa8190b9700 1 --2- 172.21.15.174:0/1791936933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8140f5410 0x7fa814106710 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fa808014000 tx=0x7fa80800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:46.063 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.061+0000 7fa805ffb700 1 -- 172.21.15.174:0/1791936933 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8080095f0 con 0x7fa8140f5410 2023-12-25T21:14:46.063 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.061+0000 7fa805ffb700 1 -- 172.21.15.174:0/1791936933 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa808007e70 con 0x7fa8140f5410 2023-12-25T21:14:46.063 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.061+0000 7fa81b31d700 1 -- 172.21.15.174:0/1791936933 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa814106e00 con 0x7fa8140f5410 2023-12-25T21:14:46.063 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.061+0000 7fa805ffb700 1 -- 172.21.15.174:0/1791936933 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa808014ea0 con 0x7fa8140f5410 2023-12-25T21:14:46.063 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.061+0000 7fa81b31d700 1 -- 172.21.15.174:0/1791936933 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa814107290 con 0x7fa8140f5410 2023-12-25T21:14:46.064 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.062+0000 7fa81b31d700 1 -- 172.21.15.174:0/1791936933 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa814045ad0 con 0x7fa8140f5410 2023-12-25T21:14:46.064 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.062+0000 7fa805ffb700 1 -- 172.21.15.174:0/1791936933 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa808025070 con 0x7fa8140f5410 2023-12-25T21:14:46.064 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.062+0000 7fa805ffb700 1 --2- 172.21.15.174:0/1791936933 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa80002ffb0 0x7fa800032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:46.064 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.062+0000 7fa805ffb700 1 -- 172.21.15.174:0/1791936933 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa808040960 con 0x7fa8140f5410 2023-12-25T21:14:46.064 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.062+0000 7fa8188b8700 1 --2- 172.21.15.174:0/1791936933 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa80002ffb0 0x7fa800032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:46.064 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.063+0000 7fa8188b8700 1 --2- 172.21.15.174:0/1791936933 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa80002ffb0 0x7fa800032470 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fa8100096a0 tx=0x7fa810006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:46.067 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.067+0000 7fa805ffb700 1 -- 172.21.15.174:0/1791936933 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8080079c0 con 0x7fa8140f5410 2023-12-25T21:14:46.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.271+0000 7fa81b31d700 1 -- 172.21.15.174:0/1791936933 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa814108140 con 0x7fa8140f5410 2023-12-25T21:14:46.272 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.272+0000 7fa805ffb700 1 -- 172.21.15.174:0/1791936933 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa80800f300 con 0x7fa8140f5410 2023-12-25T21:14:46.273 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:46.273 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:46.274 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.273+0000 7fa81b31d700 1 -- 172.21.15.174:0/1791936933 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa80002ffb0 msgr2=0x7fa800032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:46.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.273+0000 7fa81b31d700 1 --2- 172.21.15.174:0/1791936933 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa80002ffb0 0x7fa800032470 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fa8100096a0 tx=0x7fa810006b40).stop 2023-12-25T21:14:46.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.273+0000 7fa81b31d700 1 -- 172.21.15.174:0/1791936933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8140f5410 msgr2=0x7fa814106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:46.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.273+0000 7fa81b31d700 1 --2- 172.21.15.174:0/1791936933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8140f5410 0x7fa814106710 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fa808014000 tx=0x7fa80800b040).stop 2023-12-25T21:14:46.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.274+0000 7fa81b31d700 1 -- 172.21.15.174:0/1791936933 shutdown_connections 2023-12-25T21:14:46.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.274+0000 7fa81b31d700 1 --2- 172.21.15.174:0/1791936933 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fa80002ffb0 0x7fa800032470 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:46.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.274+0000 7fa81b31d700 1 --2- 172.21.15.174:0/1791936933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8140f5410 0x7fa814106710 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:46.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.274+0000 7fa81b31d700 1 -- 172.21.15.174:0/1791936933 >> 172.21.15.174:0/1791936933 conn(0x7fa8140f0b70 msgr2=0x7fa81405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:46.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.274+0000 7fa81b31d700 1 -- 172.21.15.174:0/1791936933 shutdown_connections 2023-12-25T21:14:46.275 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:46.274+0000 7fa81b31d700 1 -- 172.21.15.174:0/1791936933 wait complete. 2023-12-25T21:14:46.275 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:46.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:46 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:44.641998+0000 mgr.smithi061.jxxeke (mgr.14164) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:46.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:46 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:46.273528+0000 mon.smithi061 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.174:0/1791936933' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:47.673 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:47.673 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:48.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:48 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:46.642308+0000 mgr.smithi061.jxxeke (mgr.14164) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:48.831 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.826+0000 7efea5d4b700 1 -- 172.21.15.174:0/2722201448 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efea0060200 msgr2=0x7efea00fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:48.831 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.826+0000 7efea5d4b700 1 --2- 172.21.15.174:0/2722201448 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efea0060200 0x7efea00fb570 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7efe88009fb0 tx=0x7efe88009b30).stop 2023-12-25T21:14:48.831 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.826+0000 7efea5d4b700 1 -- 172.21.15.174:0/2722201448 shutdown_connections 2023-12-25T21:14:48.831 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.826+0000 7efea5d4b700 1 --2- 172.21.15.174:0/2722201448 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efea0060200 0x7efea00fb570 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:48.831 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.826+0000 7efea5d4b700 1 -- 172.21.15.174:0/2722201448 >> 172.21.15.174:0/2722201448 conn(0x7efea00f0b70 msgr2=0x7efea00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:48.831 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.826+0000 7efea5d4b700 1 -- 172.21.15.174:0/2722201448 shutdown_connections 2023-12-25T21:14:48.831 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.826+0000 7efea5d4b700 1 -- 172.21.15.174:0/2722201448 wait complete. 2023-12-25T21:14:48.831 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.827+0000 7efea5d4b700 1 Processor -- start 2023-12-25T21:14:48.831 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.827+0000 7efea5d4b700 1 -- start start 2023-12-25T21:14:48.831 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.827+0000 7efea5d4b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efea0060200 0x7efea01088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:48.832 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.827+0000 7efea5d4b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea0108dc0 con 0x7efea0060200 2023-12-25T21:14:48.832 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.828+0000 7efe9f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efea0060200 0x7efea01088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:48.832 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.828+0000 7efe9f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efea0060200 0x7efea01088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:45534/0 (socket says 172.21.15.174:45534) 2023-12-25T21:14:48.832 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.828+0000 7efe9f7fe700 1 -- 172.21.15.174:0/4046218780 learned_addr learned my addr 172.21.15.174:0/4046218780 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:48.832 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.828+0000 7efe9f7fe700 1 -- 172.21.15.174:0/4046218780 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe88005040 con 0x7efea0060200 2023-12-25T21:14:48.832 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.828+0000 7efe9f7fe700 1 --2- 172.21.15.174:0/4046218780 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efea0060200 0x7efea01088d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7efe88013510 tx=0x7efe8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:48.832 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.828+0000 7efe97fff700 1 -- 172.21.15.174:0/4046218780 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe88009320 con 0x7efea0060200 2023-12-25T21:14:48.832 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.828+0000 7efea5d4b700 1 -- 172.21.15.174:0/4046218780 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea0108fc0 con 0x7efea0060200 2023-12-25T21:14:48.832 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.828+0000 7efe97fff700 1 -- 172.21.15.174:0/4046218780 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe88013f00 con 0x7efea0060200 2023-12-25T21:14:48.832 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.828+0000 7efe97fff700 1 -- 172.21.15.174:0/4046218780 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe88014dc0 con 0x7efea0060200 2023-12-25T21:14:48.832 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.828+0000 7efea5d4b700 1 -- 172.21.15.174:0/4046218780 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea0109450 con 0x7efea0060200 2023-12-25T21:14:48.832 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.829+0000 7efe97fff700 1 -- 172.21.15.174:0/4046218780 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe8801d3d0 con 0x7efea0060200 2023-12-25T21:14:48.832 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.829+0000 7efe97fff700 1 --2- 172.21.15.174:0/4046218780 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efe8c02ff60 0x7efe8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:48.832 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.829+0000 7efe97fff700 1 -- 172.21.15.174:0/4046218780 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe880410e0 con 0x7efea0060200 2023-12-25T21:14:48.833 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.829+0000 7efea5d4b700 1 -- 172.21.15.174:0/4046218780 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea0045ad0 con 0x7efea0060200 2023-12-25T21:14:48.833 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.829+0000 7efe9effd700 1 --2- 172.21.15.174:0/4046218780 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efe8c02ff60 0x7efe8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:48.833 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.830+0000 7efe9effd700 1 --2- 172.21.15.174:0/4046218780 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efe8c02ff60 0x7efe8c032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7efe900096a0 tx=0x7efe90006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:48.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:48.834+0000 7efe97fff700 1 -- 172.21.15.174:0/4046218780 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe88012900 con 0x7efea0060200 2023-12-25T21:14:49.040 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:49.039+0000 7efea5d4b700 1 -- 172.21.15.174:0/4046218780 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efea010a170 con 0x7efea0060200 2023-12-25T21:14:49.041 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:49.040+0000 7efe97fff700 1 -- 172.21.15.174:0/4046218780 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efe8802b020 con 0x7efea0060200 2023-12-25T21:14:49.041 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:49.041 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:49.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:49.041+0000 7efea5d4b700 1 -- 172.21.15.174:0/4046218780 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efe8c02ff60 msgr2=0x7efe8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:49.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:49.041+0000 7efea5d4b700 1 --2- 172.21.15.174:0/4046218780 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efe8c02ff60 0x7efe8c032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7efe900096a0 tx=0x7efe90006b40).stop 2023-12-25T21:14:49.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:49.041+0000 7efea5d4b700 1 -- 172.21.15.174:0/4046218780 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efea0060200 msgr2=0x7efea01088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:49.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:49.041+0000 7efea5d4b700 1 --2- 172.21.15.174:0/4046218780 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efea0060200 0x7efea01088d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7efe88013510 tx=0x7efe8800b040).stop 2023-12-25T21:14:49.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:49.041+0000 7efea5d4b700 1 -- 172.21.15.174:0/4046218780 shutdown_connections 2023-12-25T21:14:49.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:49.041+0000 7efea5d4b700 1 --2- 172.21.15.174:0/4046218780 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efe8c02ff60 0x7efe8c032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:49.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:49.041+0000 7efea5d4b700 1 --2- 172.21.15.174:0/4046218780 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efea0060200 0x7efea01088d0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:49.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:49.041+0000 7efea5d4b700 1 -- 172.21.15.174:0/4046218780 >> 172.21.15.174:0/4046218780 conn(0x7efea00f0b70 msgr2=0x7efea00f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:49.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:49.041+0000 7efea5d4b700 1 -- 172.21.15.174:0/4046218780 shutdown_connections 2023-12-25T21:14:49.043 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:49.042+0000 7efea5d4b700 1 -- 172.21.15.174:0/4046218780 wait complete. 2023-12-25T21:14:49.043 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:49.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:49 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:49.041424+0000 mon.smithi061 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.174:0/4046218780' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:50.435 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:50.435 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:50.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:50 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:48.642535+0000 mgr.smithi061.jxxeke (mgr.14164) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:51.545 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.542+0000 7fbcbc44b700 1 -- 172.21.15.174:0/2422795311 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbcb40f5410 msgr2=0x7fbcb40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:51.545 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.542+0000 7fbcbc44b700 1 --2- 172.21.15.174:0/2422795311 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbcb40f5410 0x7fbcb40f5830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fbca4009fb0 tx=0x7fbca4009b30).stop 2023-12-25T21:14:51.545 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.542+0000 7fbcbc44b700 1 -- 172.21.15.174:0/2422795311 shutdown_connections 2023-12-25T21:14:51.545 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.542+0000 7fbcbc44b700 1 --2- 172.21.15.174:0/2422795311 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbcb40f5410 0x7fbcb40f5830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:51.545 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.542+0000 7fbcbc44b700 1 -- 172.21.15.174:0/2422795311 >> 172.21.15.174:0/2422795311 conn(0x7fbcb40f0b70 msgr2=0x7fbcb40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:51.545 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.542+0000 7fbcbc44b700 1 -- 172.21.15.174:0/2422795311 shutdown_connections 2023-12-25T21:14:51.545 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.543+0000 7fbcbc44b700 1 -- 172.21.15.174:0/2422795311 wait complete. 2023-12-25T21:14:51.545 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.543+0000 7fbcbc44b700 1 Processor -- start 2023-12-25T21:14:51.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.543+0000 7fbcbc44b700 1 -- start start 2023-12-25T21:14:51.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.543+0000 7fbcbc44b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbcb40f5410 0x7fbcb4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:51.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.543+0000 7fbcbc44b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb4105b20 con 0x7fbcb40f5410 2023-12-25T21:14:51.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.543+0000 7fbcba1e7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbcb40f5410 0x7fbcb4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:51.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.543+0000 7fbcba1e7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbcb40f5410 0x7fbcb4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:45560/0 (socket says 172.21.15.174:45560) 2023-12-25T21:14:51.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.543+0000 7fbcba1e7700 1 -- 172.21.15.174:0/529764670 learned_addr learned my addr 172.21.15.174:0/529764670 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:51.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.544+0000 7fbcba1e7700 1 -- 172.21.15.174:0/529764670 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca4005040 con 0x7fbcb40f5410 2023-12-25T21:14:51.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.544+0000 7fbcba1e7700 1 --2- 172.21.15.174:0/529764670 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbcb40f5410 0x7fbcb4105630 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fbca4013510 tx=0x7fbca400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:51.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.544+0000 7fbcaaffd700 1 -- 172.21.15.174:0/529764670 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbca4009320 con 0x7fbcb40f5410 2023-12-25T21:14:51.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.544+0000 7fbcbc44b700 1 -- 172.21.15.174:0/529764670 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb4105d20 con 0x7fbcb40f5410 2023-12-25T21:14:51.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.544+0000 7fbcaaffd700 1 -- 172.21.15.174:0/529764670 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbca4013f00 con 0x7fbcb40f5410 2023-12-25T21:14:51.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.544+0000 7fbcbc44b700 1 -- 172.21.15.174:0/529764670 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb41061b0 con 0x7fbcb40f5410 2023-12-25T21:14:51.546 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.544+0000 7fbcaaffd700 1 -- 172.21.15.174:0/529764670 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbca4014d90 con 0x7fbcb40f5410 2023-12-25T21:14:51.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.545+0000 7fbcaaffd700 1 -- 172.21.15.174:0/529764670 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbca40095f0 con 0x7fbcb40f5410 2023-12-25T21:14:51.547 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.545+0000 7fbcbc44b700 1 -- 172.21.15.174:0/529764670 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb4045ad0 con 0x7fbcb40f5410 2023-12-25T21:14:51.548 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.545+0000 7fbcaaffd700 1 --2- 172.21.15.174:0/529764670 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbca002ffb0 0x7fbca0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:51.548 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.545+0000 7fbcaaffd700 1 -- 172.21.15.174:0/529764670 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbca4041090 con 0x7fbcb40f5410 2023-12-25T21:14:51.548 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.545+0000 7fbcb99e6700 1 --2- 172.21.15.174:0/529764670 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbca002ffb0 0x7fbca0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:51.548 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.546+0000 7fbcb99e6700 1 --2- 172.21.15.174:0/529764670 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbca002ffb0 0x7fbca0032470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fbcb00096a0 tx=0x7fbcb0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:51.551 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.550+0000 7fbcaaffd700 1 -- 172.21.15.174:0/529764670 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbca406d2b0 con 0x7fbcb40f5410 2023-12-25T21:14:51.757 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.756+0000 7fbcbc44b700 1 -- 172.21.15.174:0/529764670 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbcb40fae60 con 0x7fbcb40f5410 2023-12-25T21:14:51.758 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.757+0000 7fbcaaffd700 1 -- 172.21.15.174:0/529764670 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbca4014420 con 0x7fbcb40f5410 2023-12-25T21:14:51.758 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:51.758 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:51.760 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.759+0000 7fbcbc44b700 1 -- 172.21.15.174:0/529764670 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbca002ffb0 msgr2=0x7fbca0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:51.760 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.759+0000 7fbcbc44b700 1 --2- 172.21.15.174:0/529764670 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbca002ffb0 0x7fbca0032470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fbcb00096a0 tx=0x7fbcb0006b40).stop 2023-12-25T21:14:51.760 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.759+0000 7fbcbc44b700 1 -- 172.21.15.174:0/529764670 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbcb40f5410 msgr2=0x7fbcb4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:51.760 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.759+0000 7fbcbc44b700 1 --2- 172.21.15.174:0/529764670 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbcb40f5410 0x7fbcb4105630 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fbca4013510 tx=0x7fbca400b040).stop 2023-12-25T21:14:51.760 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.759+0000 7fbcbc44b700 1 -- 172.21.15.174:0/529764670 shutdown_connections 2023-12-25T21:14:51.760 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.759+0000 7fbcbc44b700 1 --2- 172.21.15.174:0/529764670 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fbca002ffb0 0x7fbca0032470 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:51.760 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.759+0000 7fbcbc44b700 1 --2- 172.21.15.174:0/529764670 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbcb40f5410 0x7fbcb4105630 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:51.760 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.759+0000 7fbcbc44b700 1 -- 172.21.15.174:0/529764670 >> 172.21.15.174:0/529764670 conn(0x7fbcb40f0b70 msgr2=0x7fbcb40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:51.760 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.759+0000 7fbcbc44b700 1 -- 172.21.15.174:0/529764670 shutdown_connections 2023-12-25T21:14:51.761 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:51.759+0000 7fbcbc44b700 1 -- 172.21.15.174:0/529764670 wait complete. 2023-12-25T21:14:51.761 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:52.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:52 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:50.642819+0000 mgr.smithi061.jxxeke (mgr.14164) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:52.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:52 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:51.758414+0000 mon.smithi061 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.174:0/529764670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:53.146 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:53.146 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:53.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:52.709790+0000 mon.smithi061 (mon.0) 303 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/mirror_snapshot_schedule"}]: dispatch 2023-12-25T21:14:53.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:52.710132+0000 mon.smithi061 (mon.0) 304 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/trash_purge_schedule"}]: dispatch 2023-12-25T21:14:54.393 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.390+0000 7f681628c700 1 -- 172.21.15.174:0/885343900 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6810060200 msgr2=0x7f68100fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:54.393 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.390+0000 7f681628c700 1 --2- 172.21.15.174:0/885343900 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6810060200 0x7f68100fb570 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f6800009fb0 tx=0x7f6800009b30).stop 2023-12-25T21:14:54.393 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.390+0000 7f681628c700 1 -- 172.21.15.174:0/885343900 shutdown_connections 2023-12-25T21:14:54.393 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.390+0000 7f681628c700 1 --2- 172.21.15.174:0/885343900 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6810060200 0x7f68100fb570 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:54.393 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.390+0000 7f681628c700 1 -- 172.21.15.174:0/885343900 >> 172.21.15.174:0/885343900 conn(0x7f68100f0b70 msgr2=0x7f68100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:54.393 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.390+0000 7f681628c700 1 -- 172.21.15.174:0/885343900 shutdown_connections 2023-12-25T21:14:54.394 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.390+0000 7f681628c700 1 -- 172.21.15.174:0/885343900 wait complete. 2023-12-25T21:14:54.394 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.391+0000 7f681628c700 1 Processor -- start 2023-12-25T21:14:54.394 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.391+0000 7f681628c700 1 -- start start 2023-12-25T21:14:54.394 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.391+0000 7f681628c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6810060200 0x7f68101088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:54.394 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.391+0000 7f681628c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6810108dc0 con 0x7f6810060200 2023-12-25T21:14:54.394 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.391+0000 7f680ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6810060200 0x7f68101088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:54.394 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.391+0000 7f680ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6810060200 0x7f68101088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:45578/0 (socket says 172.21.15.174:45578) 2023-12-25T21:14:54.394 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.391+0000 7f680ffff700 1 -- 172.21.15.174:0/4273574266 learned_addr learned my addr 172.21.15.174:0/4273574266 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:54.394 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.392+0000 7f680ffff700 1 -- 172.21.15.174:0/4273574266 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6800005040 con 0x7f6810060200 2023-12-25T21:14:54.394 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.392+0000 7f680ffff700 1 --2- 172.21.15.174:0/4273574266 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6810060200 0x7f68101088d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f6800013510 tx=0x7f680000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:54.394 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.392+0000 7f680cff9700 1 -- 172.21.15.174:0/4273574266 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6800009320 con 0x7f6810060200 2023-12-25T21:14:54.394 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.392+0000 7f681628c700 1 -- 172.21.15.174:0/4273574266 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6810108fc0 con 0x7f6810060200 2023-12-25T21:14:54.395 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.392+0000 7f680cff9700 1 -- 172.21.15.174:0/4273574266 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6800013f00 con 0x7f6810060200 2023-12-25T21:14:54.395 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.392+0000 7f680cff9700 1 -- 172.21.15.174:0/4273574266 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6800014dc0 con 0x7f6810060200 2023-12-25T21:14:54.395 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.392+0000 7f681628c700 1 -- 172.21.15.174:0/4273574266 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6810109450 con 0x7f6810060200 2023-12-25T21:14:54.395 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.393+0000 7f680cff9700 1 -- 172.21.15.174:0/4273574266 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f680001d3d0 con 0x7f6810060200 2023-12-25T21:14:54.396 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.393+0000 7f680cff9700 1 --2- 172.21.15.174:0/4273574266 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f67f802ff60 0x7f67f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:54.396 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.393+0000 7f680cff9700 1 -- 172.21.15.174:0/4273574266 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f68000410f0 con 0x7f6810060200 2023-12-25T21:14:54.396 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.393+0000 7f681628c700 1 -- 172.21.15.174:0/4273574266 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6810045ad0 con 0x7f6810060200 2023-12-25T21:14:54.396 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.393+0000 7f680f7fe700 1 --2- 172.21.15.174:0/4273574266 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f67f802ff60 0x7f67f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:54.396 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.394+0000 7f680f7fe700 1 --2- 172.21.15.174:0/4273574266 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f67f802ff60 0x7f67f8032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f68040096a0 tx=0x7f6804006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:54.399 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.398+0000 7f680cff9700 1 -- 172.21.15.174:0/4273574266 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6800012920 con 0x7f6810060200 2023-12-25T21:14:54.604 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.604+0000 7f681628c700 1 -- 172.21.15.174:0/4273574266 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f681010a130 con 0x7f6810060200 2023-12-25T21:14:54.605 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.604+0000 7f680cff9700 1 -- 172.21.15.174:0/4273574266 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6800012330 con 0x7f6810060200 2023-12-25T21:14:54.605 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:54.606 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:54.607 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.606+0000 7f681628c700 1 -- 172.21.15.174:0/4273574266 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f67f802ff60 msgr2=0x7f67f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:54.607 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.606+0000 7f681628c700 1 --2- 172.21.15.174:0/4273574266 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f67f802ff60 0x7f67f8032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f68040096a0 tx=0x7f6804006b40).stop 2023-12-25T21:14:54.607 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.606+0000 7f681628c700 1 -- 172.21.15.174:0/4273574266 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6810060200 msgr2=0x7f68101088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:54.607 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.606+0000 7f681628c700 1 --2- 172.21.15.174:0/4273574266 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6810060200 0x7f68101088d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f6800013510 tx=0x7f680000b040).stop 2023-12-25T21:14:54.607 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.606+0000 7f681628c700 1 -- 172.21.15.174:0/4273574266 shutdown_connections 2023-12-25T21:14:54.607 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.606+0000 7f681628c700 1 --2- 172.21.15.174:0/4273574266 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f67f802ff60 0x7f67f8032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:54.607 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.606+0000 7f681628c700 1 --2- 172.21.15.174:0/4273574266 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6810060200 0x7f68101088d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:54.607 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.606+0000 7f681628c700 1 -- 172.21.15.174:0/4273574266 >> 172.21.15.174:0/4273574266 conn(0x7f68100f0b70 msgr2=0x7f68100f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:54.608 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.606+0000 7f681628c700 1 -- 172.21.15.174:0/4273574266 shutdown_connections 2023-12-25T21:14:54.608 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:54.606+0000 7f681628c700 1 -- 172.21.15.174:0/4273574266 wait complete. 2023-12-25T21:14:54.608 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:54.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:54 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:52.643222+0000 mgr.smithi061.jxxeke (mgr.14164) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:55.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:55 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:54.605913+0000 mon.smithi061 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.174:0/4273574266' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:56.012 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:56.012 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:56.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:56 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:54.643462+0000 mgr.smithi061.jxxeke (mgr.14164) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:57.148 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.145+0000 7f7ffed3b700 1 -- 172.21.15.174:0/2549916606 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff80f5810 msgr2=0x7f7ff80f5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:57.148 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.145+0000 7f7ffed3b700 1 --2- 172.21.15.174:0/2549916606 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff80f5810 0x7f7ff80f5c30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f7fec009fb0 tx=0x7f7fec009b30).stop 2023-12-25T21:14:57.148 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.145+0000 7f7ffed3b700 1 -- 172.21.15.174:0/2549916606 shutdown_connections 2023-12-25T21:14:57.148 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.145+0000 7f7ffed3b700 1 --2- 172.21.15.174:0/2549916606 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff80f5810 0x7f7ff80f5c30 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:57.148 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.145+0000 7f7ffed3b700 1 -- 172.21.15.174:0/2549916606 >> 172.21.15.174:0/2549916606 conn(0x7f7ff80f0120 msgr2=0x7f7ff80f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:57.148 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.146+0000 7f7ffed3b700 1 -- 172.21.15.174:0/2549916606 shutdown_connections 2023-12-25T21:14:57.148 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.146+0000 7f7ffed3b700 1 -- 172.21.15.174:0/2549916606 wait complete. 2023-12-25T21:14:57.148 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.146+0000 7f7ffed3b700 1 Processor -- start 2023-12-25T21:14:57.148 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.146+0000 7f7ffed3b700 1 -- start start 2023-12-25T21:14:57.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.146+0000 7f7ffed3b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff80f5810 0x7f7ff81086e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:57.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.146+0000 7f7ffed3b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff8108bd0 con 0x7f7ff80f5810 2023-12-25T21:14:57.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.146+0000 7f7ffdd39700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff80f5810 0x7f7ff81086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:57.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.146+0000 7f7ffdd39700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff80f5810 0x7f7ff81086e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:48302/0 (socket says 172.21.15.174:48302) 2023-12-25T21:14:57.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.146+0000 7f7ffdd39700 1 -- 172.21.15.174:0/2819679884 learned_addr learned my addr 172.21.15.174:0/2819679884 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:57.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.147+0000 7f7ffdd39700 1 -- 172.21.15.174:0/2819679884 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fec005040 con 0x7f7ff80f5810 2023-12-25T21:14:57.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.147+0000 7f7ffdd39700 1 --2- 172.21.15.174:0/2819679884 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff80f5810 0x7f7ff81086e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f7fec004dd0 tx=0x7f7fec009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:57.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.147+0000 7f7fea7fc700 1 -- 172.21.15.174:0/2819679884 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fec018070 con 0x7f7ff80f5810 2023-12-25T21:14:57.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.147+0000 7f7ffed3b700 1 -- 172.21.15.174:0/2819679884 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff8108dd0 con 0x7f7ff80f5810 2023-12-25T21:14:57.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.147+0000 7f7fea7fc700 1 -- 172.21.15.174:0/2819679884 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7fec013410 con 0x7f7ff80f5810 2023-12-25T21:14:57.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.147+0000 7f7fea7fc700 1 -- 172.21.15.174:0/2819679884 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fec01c3c0 con 0x7f7ff80f5810 2023-12-25T21:14:57.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.147+0000 7f7ffed3b700 1 -- 172.21.15.174:0/2819679884 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff8109260 con 0x7f7ff80f5810 2023-12-25T21:14:57.150 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.148+0000 7f7fea7fc700 1 -- 172.21.15.174:0/2819679884 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7fec01d420 con 0x7f7ff80f5810 2023-12-25T21:14:57.150 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.148+0000 7f7ffed3b700 1 -- 172.21.15.174:0/2819679884 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff8045ad0 con 0x7f7ff80f5810 2023-12-25T21:14:57.150 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.148+0000 7f7fea7fc700 1 --2- 172.21.15.174:0/2819679884 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7fe402ff60 0x7f7fe4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:57.150 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.148+0000 7f7fea7fc700 1 -- 172.21.15.174:0/2819679884 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7fec0413f0 con 0x7f7ff80f5810 2023-12-25T21:14:57.150 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.148+0000 7f7ffd538700 1 --2- 172.21.15.174:0/2819679884 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7fe402ff60 0x7f7fe4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:57.151 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.149+0000 7f7ffd538700 1 --2- 172.21.15.174:0/2819679884 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7fe402ff60 0x7f7fe4032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f7ff40096a0 tx=0x7f7ff4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:57.153 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.153+0000 7f7fea7fc700 1 -- 172.21.15.174:0/2819679884 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7fec01d820 con 0x7f7ff80f5810 2023-12-25T21:14:57.360 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.359+0000 7f7ffed3b700 1 -- 172.21.15.174:0/2819679884 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ff80fabf0 con 0x7f7ff80f5810 2023-12-25T21:14:57.360 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.360+0000 7f7fea7fc700 1 -- 172.21.15.174:0/2819679884 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7fec01b370 con 0x7f7ff80f5810 2023-12-25T21:14:57.361 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:14:57.361 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:14:57.362 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.361+0000 7f7ffed3b700 1 -- 172.21.15.174:0/2819679884 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7fe402ff60 msgr2=0x7f7fe4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:57.362 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.361+0000 7f7ffed3b700 1 --2- 172.21.15.174:0/2819679884 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7fe402ff60 0x7f7fe4032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f7ff40096a0 tx=0x7f7ff4006b40).stop 2023-12-25T21:14:57.362 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.361+0000 7f7ffed3b700 1 -- 172.21.15.174:0/2819679884 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff80f5810 msgr2=0x7f7ff81086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:57.362 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.361+0000 7f7ffed3b700 1 --2- 172.21.15.174:0/2819679884 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff80f5810 0x7f7ff81086e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f7fec004dd0 tx=0x7f7fec009b30).stop 2023-12-25T21:14:57.363 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.361+0000 7f7ffed3b700 1 -- 172.21.15.174:0/2819679884 shutdown_connections 2023-12-25T21:14:57.363 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.361+0000 7f7ffed3b700 1 --2- 172.21.15.174:0/2819679884 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7fe402ff60 0x7f7fe4032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:57.363 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.361+0000 7f7ffed3b700 1 --2- 172.21.15.174:0/2819679884 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff80f5810 0x7f7ff81086e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:57.363 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.361+0000 7f7ffed3b700 1 -- 172.21.15.174:0/2819679884 >> 172.21.15.174:0/2819679884 conn(0x7f7ff80f0120 msgr2=0x7f7ff80f4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:57.363 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.362+0000 7f7ffed3b700 1 -- 172.21.15.174:0/2819679884 shutdown_connections 2023-12-25T21:14:57.363 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:57.362+0000 7f7ffed3b700 1 -- 172.21.15.174:0/2819679884 wait complete. 2023-12-25T21:14:57.363 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:14:57.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:57 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:14:57.361420+0000 mon.smithi061 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.174:0/2819679884' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:14:58.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:14:58 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:56.643747+0000 mgr.smithi061.jxxeke (mgr.14164) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:14:58.758 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:14:58.759 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:14:59.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.964+0000 7f51d140e700 1 -- 172.21.15.174:0/2987791974 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51cc0f5410 msgr2=0x7f51cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:14:59.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.964+0000 7f51d140e700 1 --2- 172.21.15.174:0/2987791974 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51cc0f5410 0x7f51cc0f5830 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f51bc009fb0 tx=0x7f51bc009b30).stop 2023-12-25T21:14:59.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.964+0000 7f51d140e700 1 -- 172.21.15.174:0/2987791974 shutdown_connections 2023-12-25T21:14:59.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.964+0000 7f51d140e700 1 --2- 172.21.15.174:0/2987791974 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51cc0f5410 0x7f51cc0f5830 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:14:59.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.964+0000 7f51d140e700 1 -- 172.21.15.174:0/2987791974 >> 172.21.15.174:0/2987791974 conn(0x7f51cc0f0b70 msgr2=0x7f51cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:14:59.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.964+0000 7f51d140e700 1 -- 172.21.15.174:0/2987791974 shutdown_connections 2023-12-25T21:14:59.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.964+0000 7f51d140e700 1 -- 172.21.15.174:0/2987791974 wait complete. 2023-12-25T21:14:59.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.965+0000 7f51d140e700 1 Processor -- start 2023-12-25T21:14:59.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.965+0000 7f51d140e700 1 -- start start 2023-12-25T21:14:59.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.965+0000 7f51d140e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51cc0f5410 0x7f51cc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:59.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.965+0000 7f51d140e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc106c20 con 0x7f51cc0f5410 2023-12-25T21:14:59.968 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.965+0000 7f51caffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51cc0f5410 0x7f51cc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:59.968 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.965+0000 7f51caffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51cc0f5410 0x7f51cc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:48332/0 (socket says 172.21.15.174:48332) 2023-12-25T21:14:59.968 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.965+0000 7f51caffd700 1 -- 172.21.15.174:0/1406158758 learned_addr learned my addr 172.21.15.174:0/1406158758 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:14:59.969 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.966+0000 7f51caffd700 1 -- 172.21.15.174:0/1406158758 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51bc005040 con 0x7f51cc0f5410 2023-12-25T21:14:59.969 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.966+0000 7f51caffd700 1 --2- 172.21.15.174:0/1406158758 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51cc0f5410 0x7f51cc106730 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f51bc013510 tx=0x7f51bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:59.969 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.966+0000 7f51b37fe700 1 -- 172.21.15.174:0/1406158758 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51bc009320 con 0x7f51cc0f5410 2023-12-25T21:14:59.969 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.966+0000 7f51d140e700 1 -- 172.21.15.174:0/1406158758 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51cc106e20 con 0x7f51cc0f5410 2023-12-25T21:14:59.969 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.966+0000 7f51b37fe700 1 -- 172.21.15.174:0/1406158758 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f51bc013f00 con 0x7f51cc0f5410 2023-12-25T21:14:59.969 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.966+0000 7f51b37fe700 1 -- 172.21.15.174:0/1406158758 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51bc014d90 con 0x7f51cc0f5410 2023-12-25T21:14:59.969 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.966+0000 7f51d140e700 1 -- 172.21.15.174:0/1406158758 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51cc1072b0 con 0x7f51cc0f5410 2023-12-25T21:14:59.969 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.967+0000 7f51b37fe700 1 -- 172.21.15.174:0/1406158758 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51bc0263a0 con 0x7f51cc0f5410 2023-12-25T21:14:59.969 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.967+0000 7f51d140e700 1 -- 172.21.15.174:0/1406158758 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51cc045ad0 con 0x7f51cc0f5410 2023-12-25T21:14:59.969 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.967+0000 7f51b37fe700 1 --2- 172.21.15.174:0/1406158758 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f51b402ff60 0x7f51b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:14:59.969 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.967+0000 7f51b37fe700 1 -- 172.21.15.174:0/1406158758 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f51bc040ff0 con 0x7f51cc0f5410 2023-12-25T21:14:59.969 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.967+0000 7f51ca7fc700 1 --2- 172.21.15.174:0/1406158758 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f51b402ff60 0x7f51b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:14:59.970 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.968+0000 7f51ca7fc700 1 --2- 172.21.15.174:0/1406158758 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f51b402ff60 0x7f51b4032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f51c00096a0 tx=0x7f51c0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:14:59.972 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:14:59.973+0000 7f51b37fe700 1 -- 172.21.15.174:0/1406158758 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51bc0095f0 con 0x7f51cc0f5410 2023-12-25T21:15:00.177 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:00.176+0000 7f51d140e700 1 -- 172.21.15.174:0/1406158758 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51cc0fae60 con 0x7f51cc0f5410 2023-12-25T21:15:00.178 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:00.177+0000 7f51b37fe700 1 -- 172.21.15.174:0/1406158758 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f51bc01db70 con 0x7f51cc0f5410 2023-12-25T21:15:00.178 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:00.178 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:00.180 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:00.179+0000 7f51d140e700 1 -- 172.21.15.174:0/1406158758 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f51b402ff60 msgr2=0x7f51b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:00.180 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:00.179+0000 7f51d140e700 1 --2- 172.21.15.174:0/1406158758 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f51b402ff60 0x7f51b4032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f51c00096a0 tx=0x7f51c0006b40).stop 2023-12-25T21:15:00.180 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:00.179+0000 7f51d140e700 1 -- 172.21.15.174:0/1406158758 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51cc0f5410 msgr2=0x7f51cc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:00.180 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:00.179+0000 7f51d140e700 1 --2- 172.21.15.174:0/1406158758 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51cc0f5410 0x7f51cc106730 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f51bc013510 tx=0x7f51bc00b040).stop 2023-12-25T21:15:00.180 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:00.179+0000 7f51d140e700 1 -- 172.21.15.174:0/1406158758 shutdown_connections 2023-12-25T21:15:00.180 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:00.179+0000 7f51d140e700 1 --2- 172.21.15.174:0/1406158758 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f51b402ff60 0x7f51b4032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:00.180 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:00.179+0000 7f51d140e700 1 --2- 172.21.15.174:0/1406158758 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51cc0f5410 0x7f51cc106730 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:00.180 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:00.179+0000 7f51d140e700 1 -- 172.21.15.174:0/1406158758 >> 172.21.15.174:0/1406158758 conn(0x7f51cc0f0b70 msgr2=0x7f51cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:00.180 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:00.179+0000 7f51d140e700 1 -- 172.21.15.174:0/1406158758 shutdown_connections 2023-12-25T21:15:00.180 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:00.179+0000 7f51d140e700 1 -- 172.21.15.174:0/1406158758 wait complete. 2023-12-25T21:15:00.181 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:00.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:00 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:14:58.644114+0000 mgr.smithi061.jxxeke (mgr.14164) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:00.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:00 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:00.178542+0000 mon.smithi061 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.174:0/1406158758' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:15:01.601 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:15:01.602 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:15:02.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:02 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:00.644474+0000 mgr.smithi061.jxxeke (mgr.14164) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:02.767 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.767+0000 7f0265515700 1 -- 172.21.15.174:0/3466407423 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026006b960 msgr2=0x7f0260069e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:02.767 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.767+0000 7f0265515700 1 --2- 172.21.15.174:0/3466407423 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026006b960 0x7f0260069e60 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f0254009fb0 tx=0x7f0254009b30).stop 2023-12-25T21:15:02.767 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.767+0000 7f0265515700 1 -- 172.21.15.174:0/3466407423 shutdown_connections 2023-12-25T21:15:02.767 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.767+0000 7f0265515700 1 --2- 172.21.15.174:0/3466407423 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026006b960 0x7f0260069e60 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:02.767 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.767+0000 7f0265515700 1 -- 172.21.15.174:0/3466407423 >> 172.21.15.174:0/3466407423 conn(0x7f02600f1c70 msgr2=0x7f02600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:02.767 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.767+0000 7f0265515700 1 -- 172.21.15.174:0/3466407423 shutdown_connections 2023-12-25T21:15:02.767 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.767+0000 7f0265515700 1 -- 172.21.15.174:0/3466407423 wait complete. 2023-12-25T21:15:02.769 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.767+0000 7f0265515700 1 Processor -- start 2023-12-25T21:15:02.769 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.768+0000 7f0265515700 1 -- start start 2023-12-25T21:15:02.769 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.768+0000 7f0265515700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026006b960 0x7f02601088c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:02.769 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.768+0000 7f0265515700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0260108db0 con 0x7f026006b960 2023-12-25T21:15:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.768+0000 7f025effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026006b960 0x7f02601088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.768+0000 7f025effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026006b960 0x7f02601088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:48358/0 (socket says 172.21.15.174:48358) 2023-12-25T21:15:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.768+0000 7f025effd700 1 -- 172.21.15.174:0/3538851128 learned_addr learned my addr 172.21.15.174:0/3538851128 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:15:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.768+0000 7f025effd700 1 -- 172.21.15.174:0/3538851128 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0254005040 con 0x7f026006b960 2023-12-25T21:15:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.768+0000 7f025effd700 1 --2- 172.21.15.174:0/3538851128 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026006b960 0x7f02601088c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f0254013510 tx=0x7f025400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.768+0000 7f02477fe700 1 -- 172.21.15.174:0/3538851128 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0254009320 con 0x7f026006b960 2023-12-25T21:15:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.769+0000 7f0265515700 1 -- 172.21.15.174:0/3538851128 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0260108fb0 con 0x7f026006b960 2023-12-25T21:15:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.769+0000 7f02477fe700 1 -- 172.21.15.174:0/3538851128 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0254013f00 con 0x7f026006b960 2023-12-25T21:15:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.769+0000 7f02477fe700 1 -- 172.21.15.174:0/3538851128 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0254014dc0 con 0x7f026006b960 2023-12-25T21:15:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.769+0000 7f0265515700 1 -- 172.21.15.174:0/3538851128 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0260109440 con 0x7f026006b960 2023-12-25T21:15:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.769+0000 7f02477fe700 1 -- 172.21.15.174:0/3538851128 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f025401d3d0 con 0x7f026006b960 2023-12-25T21:15:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.769+0000 7f0265515700 1 -- 172.21.15.174:0/3538851128 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0260045ad0 con 0x7f026006b960 2023-12-25T21:15:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.769+0000 7f02477fe700 1 --2- 172.21.15.174:0/3538851128 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f024802ff60 0x7f0248032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:02.770 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.769+0000 7f02477fe700 1 -- 172.21.15.174:0/3538851128 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f02540410f0 con 0x7f026006b960 2023-12-25T21:15:02.771 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.770+0000 7f025e7fc700 1 --2- 172.21.15.174:0/3538851128 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f024802ff60 0x7f0248032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:02.771 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.770+0000 7f025e7fc700 1 --2- 172.21.15.174:0/3538851128 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f024802ff60 0x7f0248032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f02500096a0 tx=0x7f0250006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:02.774 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.775+0000 7f02477fe700 1 -- 172.21.15.174:0/3538851128 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0254012920 con 0x7f026006b960 2023-12-25T21:15:02.980 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.980+0000 7f0265515700 1 -- 172.21.15.174:0/3538851128 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02600fe150 con 0x7f026006b960 2023-12-25T21:15:02.981 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.981+0000 7f02477fe700 1 -- 172.21.15.174:0/3538851128 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f02540126f0 con 0x7f026006b960 2023-12-25T21:15:02.981 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:02.981 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:02.982 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.982+0000 7f0265515700 1 -- 172.21.15.174:0/3538851128 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f024802ff60 msgr2=0x7f0248032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:02.982 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.982+0000 7f0265515700 1 --2- 172.21.15.174:0/3538851128 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f024802ff60 0x7f0248032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f02500096a0 tx=0x7f0250006b40).stop 2023-12-25T21:15:02.982 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.982+0000 7f0265515700 1 -- 172.21.15.174:0/3538851128 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026006b960 msgr2=0x7f02601088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:02.982 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.982+0000 7f0265515700 1 --2- 172.21.15.174:0/3538851128 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026006b960 0x7f02601088c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f0254013510 tx=0x7f025400b040).stop 2023-12-25T21:15:02.982 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.982+0000 7f0265515700 1 -- 172.21.15.174:0/3538851128 shutdown_connections 2023-12-25T21:15:02.982 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.982+0000 7f0265515700 1 --2- 172.21.15.174:0/3538851128 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f024802ff60 0x7f0248032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:02.983 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.982+0000 7f0265515700 1 --2- 172.21.15.174:0/3538851128 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026006b960 0x7f02601088c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:02.983 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.982+0000 7f0265515700 1 -- 172.21.15.174:0/3538851128 >> 172.21.15.174:0/3538851128 conn(0x7f02600f1c70 msgr2=0x7f02600f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:02.983 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.982+0000 7f0265515700 1 -- 172.21.15.174:0/3538851128 shutdown_connections 2023-12-25T21:15:02.983 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:02.982+0000 7f0265515700 1 -- 172.21.15.174:0/3538851128 wait complete. 2023-12-25T21:15:02.983 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:03.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:03 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:02.981236+0000 mon.smithi061 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.174:0/3538851128' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:15:04.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:15:04.367 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:15:04.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:04 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:02.644798+0000 mgr.smithi061.jxxeke (mgr.14164) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:05.523 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.522+0000 7f6d5c99f700 1 -- 172.21.15.174:0/1259833825 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d58060280 msgr2=0x7f6d580f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:05.524 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.522+0000 7f6d5c99f700 1 --2- 172.21.15.174:0/1259833825 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d58060280 0x7f6d580f90f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f6d4c009fb0 tx=0x7f6d4c009b30).stop 2023-12-25T21:15:05.524 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.522+0000 7f6d5c99f700 1 -- 172.21.15.174:0/1259833825 shutdown_connections 2023-12-25T21:15:05.524 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.522+0000 7f6d5c99f700 1 --2- 172.21.15.174:0/1259833825 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d58060280 0x7f6d580f90f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:05.524 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.522+0000 7f6d5c99f700 1 -- 172.21.15.174:0/1259833825 >> 172.21.15.174:0/1259833825 conn(0x7f6d580f0b70 msgr2=0x7f6d580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:05.524 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.522+0000 7f6d5c99f700 1 -- 172.21.15.174:0/1259833825 shutdown_connections 2023-12-25T21:15:05.524 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.522+0000 7f6d5c99f700 1 -- 172.21.15.174:0/1259833825 wait complete. 2023-12-25T21:15:05.524 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.523+0000 7f6d5c99f700 1 Processor -- start 2023-12-25T21:15:05.524 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.523+0000 7f6d5c99f700 1 -- start start 2023-12-25T21:15:05.524 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.523+0000 7f6d5c99f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d58060280 0x7f6d581066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:05.524 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.523+0000 7f6d5c99f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d58106bd0 con 0x7f6d58060280 2023-12-25T21:15:05.524 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.523+0000 7f6d5659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d58060280 0x7f6d581066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:05.525 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.523+0000 7f6d5659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d58060280 0x7f6d581066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:48374/0 (socket says 172.21.15.174:48374) 2023-12-25T21:15:05.525 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.523+0000 7f6d5659c700 1 -- 172.21.15.174:0/2611305493 learned_addr learned my addr 172.21.15.174:0/2611305493 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:15:05.525 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.524+0000 7f6d5659c700 1 -- 172.21.15.174:0/2611305493 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4c005040 con 0x7f6d58060280 2023-12-25T21:15:05.525 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.524+0000 7f6d5659c700 1 --2- 172.21.15.174:0/2611305493 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d58060280 0x7f6d581066e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f6d4c013510 tx=0x7f6d4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:05.525 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.524+0000 7f6d46ffd700 1 -- 172.21.15.174:0/2611305493 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d4c009320 con 0x7f6d58060280 2023-12-25T21:15:05.525 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.524+0000 7f6d5c99f700 1 -- 172.21.15.174:0/2611305493 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d58106dd0 con 0x7f6d58060280 2023-12-25T21:15:05.525 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.524+0000 7f6d46ffd700 1 -- 172.21.15.174:0/2611305493 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d4c013f00 con 0x7f6d58060280 2023-12-25T21:15:05.525 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.524+0000 7f6d46ffd700 1 -- 172.21.15.174:0/2611305493 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d4c014dc0 con 0x7f6d58060280 2023-12-25T21:15:05.525 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.524+0000 7f6d5c99f700 1 -- 172.21.15.174:0/2611305493 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d58107260 con 0x7f6d58060280 2023-12-25T21:15:05.526 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.525+0000 7f6d46ffd700 1 -- 172.21.15.174:0/2611305493 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d4c01d3d0 con 0x7f6d58060280 2023-12-25T21:15:05.526 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.525+0000 7f6d46ffd700 1 --2- 172.21.15.174:0/2611305493 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6d4002ff60 0x7f6d40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:05.526 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.525+0000 7f6d46ffd700 1 -- 172.21.15.174:0/2611305493 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d4c0410a0 con 0x7f6d58060280 2023-12-25T21:15:05.526 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.525+0000 7f6d5c99f700 1 -- 172.21.15.174:0/2611305493 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d58045ad0 con 0x7f6d58060280 2023-12-25T21:15:05.526 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.525+0000 7f6d55d9b700 1 --2- 172.21.15.174:0/2611305493 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6d4002ff60 0x7f6d40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:05.526 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.525+0000 7f6d55d9b700 1 --2- 172.21.15.174:0/2611305493 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6d4002ff60 0x7f6d40032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f6d480096a0 tx=0x7f6d48006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:05.530 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.530+0000 7f6d46ffd700 1 -- 172.21.15.174:0/2611305493 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d4c012920 con 0x7f6d58060280 2023-12-25T21:15:05.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:05 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:04.645098+0000 mgr.smithi061.jxxeke (mgr.14164) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:05.737 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.738+0000 7f6d5c99f700 1 -- 172.21.15.174:0/2611305493 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d580fc040 con 0x7f6d58060280 2023-12-25T21:15:05.738 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.738+0000 7f6d46ffd700 1 -- 172.21.15.174:0/2611305493 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6d4c02b020 con 0x7f6d58060280 2023-12-25T21:15:05.738 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:05.738 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:05.739 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.740+0000 7f6d5c99f700 1 -- 172.21.15.174:0/2611305493 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6d4002ff60 msgr2=0x7f6d40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:05.740 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.740+0000 7f6d5c99f700 1 --2- 172.21.15.174:0/2611305493 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6d4002ff60 0x7f6d40032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f6d480096a0 tx=0x7f6d48006b40).stop 2023-12-25T21:15:05.740 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.740+0000 7f6d5c99f700 1 -- 172.21.15.174:0/2611305493 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d58060280 msgr2=0x7f6d581066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:05.740 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.740+0000 7f6d5c99f700 1 --2- 172.21.15.174:0/2611305493 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d58060280 0x7f6d581066e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f6d4c013510 tx=0x7f6d4c00b040).stop 2023-12-25T21:15:05.740 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.740+0000 7f6d5c99f700 1 -- 172.21.15.174:0/2611305493 shutdown_connections 2023-12-25T21:15:05.740 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.740+0000 7f6d5c99f700 1 --2- 172.21.15.174:0/2611305493 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f6d4002ff60 0x7f6d40032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:05.740 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.740+0000 7f6d5c99f700 1 --2- 172.21.15.174:0/2611305493 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d58060280 0x7f6d581066e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:05.740 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.740+0000 7f6d5c99f700 1 -- 172.21.15.174:0/2611305493 >> 172.21.15.174:0/2611305493 conn(0x7f6d580f0b70 msgr2=0x7f6d580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:05.740 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.740+0000 7f6d5c99f700 1 -- 172.21.15.174:0/2611305493 shutdown_connections 2023-12-25T21:15:05.740 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:05.740+0000 7f6d5c99f700 1 -- 172.21.15.174:0/2611305493 wait complete. 2023-12-25T21:15:05.740 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:06.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:06 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:05.738806+0000 mon.smithi061 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.174:0/2611305493' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:15:07.131 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:15:07.131 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:15:07.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:07 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:06.645430+0000 mgr.smithi061.jxxeke (mgr.14164) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:08.357 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.356+0000 7fb8269ba700 1 -- 172.21.15.174:0/2822091750 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8200601e0 msgr2=0x7fb8200f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:08.357 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.356+0000 7fb8269ba700 1 --2- 172.21.15.174:0/2822091750 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8200601e0 0x7fb8200f8f20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fb81c009fb0 tx=0x7fb81c009b30).stop 2023-12-25T21:15:08.357 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.356+0000 7fb8269ba700 1 -- 172.21.15.174:0/2822091750 shutdown_connections 2023-12-25T21:15:08.357 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.356+0000 7fb8269ba700 1 --2- 172.21.15.174:0/2822091750 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8200601e0 0x7fb8200f8f20 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:08.357 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.356+0000 7fb8269ba700 1 -- 172.21.15.174:0/2822091750 >> 172.21.15.174:0/2822091750 conn(0x7fb8200f09a0 msgr2=0x7fb8200f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:08.358 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.356+0000 7fb8269ba700 1 -- 172.21.15.174:0/2822091750 shutdown_connections 2023-12-25T21:15:08.358 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.356+0000 7fb8269ba700 1 -- 172.21.15.174:0/2822091750 wait complete. 2023-12-25T21:15:08.358 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.357+0000 7fb8269ba700 1 Processor -- start 2023-12-25T21:15:08.358 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.357+0000 7fb8269ba700 1 -- start start 2023-12-25T21:15:08.358 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.357+0000 7fb8269ba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8200601e0 0x7fb820106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:08.358 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.357+0000 7fb8269ba700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8201069f0 con 0x7fb8200601e0 2023-12-25T21:15:08.358 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.357+0000 7fb8259b8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8200601e0 0x7fb820106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:08.358 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.357+0000 7fb8259b8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8200601e0 0x7fb820106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:54248/0 (socket says 172.21.15.174:54248) 2023-12-25T21:15:08.358 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.357+0000 7fb8259b8700 1 -- 172.21.15.174:0/2148071799 learned_addr learned my addr 172.21.15.174:0/2148071799 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:15:08.360 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.358+0000 7fb8259b8700 1 -- 172.21.15.174:0/2148071799 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81c005040 con 0x7fb8200601e0 2023-12-25T21:15:08.360 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.358+0000 7fb8259b8700 1 --2- 172.21.15.174:0/2148071799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8200601e0 0x7fb820106500 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fb81c013510 tx=0x7fb81c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:08.360 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.358+0000 7fb8167fc700 1 -- 172.21.15.174:0/2148071799 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb81c009320 con 0x7fb8200601e0 2023-12-25T21:15:08.360 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.358+0000 7fb8269ba700 1 -- 172.21.15.174:0/2148071799 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb820106bf0 con 0x7fb8200601e0 2023-12-25T21:15:08.360 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.358+0000 7fb8167fc700 1 -- 172.21.15.174:0/2148071799 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb81c013f00 con 0x7fb8200601e0 2023-12-25T21:15:08.360 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.358+0000 7fb8167fc700 1 -- 172.21.15.174:0/2148071799 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb81c014d90 con 0x7fb8200601e0 2023-12-25T21:15:08.360 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.358+0000 7fb8269ba700 1 -- 172.21.15.174:0/2148071799 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb820107080 con 0x7fb8200601e0 2023-12-25T21:15:08.360 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.359+0000 7fb8167fc700 1 -- 172.21.15.174:0/2148071799 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb81c0095f0 con 0x7fb8200601e0 2023-12-25T21:15:08.360 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.359+0000 7fb8269ba700 1 -- 172.21.15.174:0/2148071799 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb820045ad0 con 0x7fb8200601e0 2023-12-25T21:15:08.360 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.359+0000 7fb8167fc700 1 --2- 172.21.15.174:0/2148071799 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb80c02ff60 0x7fb80c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:08.360 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.359+0000 7fb8167fc700 1 -- 172.21.15.174:0/2148071799 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb81c0410c0 con 0x7fb8200601e0 2023-12-25T21:15:08.360 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.359+0000 7fb8251b7700 1 --2- 172.21.15.174:0/2148071799 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb80c02ff60 0x7fb80c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:08.361 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.359+0000 7fb8251b7700 1 --2- 172.21.15.174:0/2148071799 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb80c02ff60 0x7fb80c032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fb8100096a0 tx=0x7fb810006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:08.364 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.364+0000 7fb8167fc700 1 -- 172.21.15.174:0/2148071799 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb81c014420 con 0x7fb8200601e0 2023-12-25T21:15:08.571 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.571+0000 7fb8269ba700 1 -- 172.21.15.174:0/2148071799 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb820107f30 con 0x7fb8200601e0 2023-12-25T21:15:08.572 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.572+0000 7fb8167fc700 1 -- 172.21.15.174:0/2148071799 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb81c010230 con 0x7fb8200601e0 2023-12-25T21:15:08.572 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:08.573 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:08.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.574+0000 7fb8269ba700 1 -- 172.21.15.174:0/2148071799 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb80c02ff60 msgr2=0x7fb80c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:08.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.574+0000 7fb8269ba700 1 --2- 172.21.15.174:0/2148071799 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb80c02ff60 0x7fb80c032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fb8100096a0 tx=0x7fb810006b40).stop 2023-12-25T21:15:08.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.574+0000 7fb8269ba700 1 -- 172.21.15.174:0/2148071799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8200601e0 msgr2=0x7fb820106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:08.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.574+0000 7fb8269ba700 1 --2- 172.21.15.174:0/2148071799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8200601e0 0x7fb820106500 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fb81c013510 tx=0x7fb81c00b040).stop 2023-12-25T21:15:08.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.574+0000 7fb8269ba700 1 -- 172.21.15.174:0/2148071799 shutdown_connections 2023-12-25T21:15:08.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.574+0000 7fb8269ba700 1 --2- 172.21.15.174:0/2148071799 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb80c02ff60 0x7fb80c032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:08.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.574+0000 7fb8269ba700 1 --2- 172.21.15.174:0/2148071799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8200601e0 0x7fb820106500 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:08.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.574+0000 7fb8269ba700 1 -- 172.21.15.174:0/2148071799 >> 172.21.15.174:0/2148071799 conn(0x7fb8200f09a0 msgr2=0x7fb8200f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:08.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.574+0000 7fb8269ba700 1 -- 172.21.15.174:0/2148071799 shutdown_connections 2023-12-25T21:15:08.574 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:08.574+0000 7fb8269ba700 1 -- 172.21.15.174:0/2148071799 wait complete. 2023-12-25T21:15:08.575 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:08.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:08 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:08.572805+0000 mon.smithi061 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.174:0/2148071799' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:15:09.929 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:15:09.930 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:15:09.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:09 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:08.645741+0000 mgr.smithi061.jxxeke (mgr.14164) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:11.041 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.041+0000 7f693bfff700 1 -- 172.21.15.174:0/1034882718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f693c0f51a0 msgr2=0x7f693c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:11.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.041+0000 7f693bfff700 1 --2- 172.21.15.174:0/1034882718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f693c0f51a0 0x7f693c0f55c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f692c009fb0 tx=0x7f692c009b30).stop 2023-12-25T21:15:11.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.041+0000 7f693bfff700 1 -- 172.21.15.174:0/1034882718 shutdown_connections 2023-12-25T21:15:11.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.041+0000 7f693bfff700 1 --2- 172.21.15.174:0/1034882718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f693c0f51a0 0x7f693c0f55c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:11.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.041+0000 7f693bfff700 1 -- 172.21.15.174:0/1034882718 >> 172.21.15.174:0/1034882718 conn(0x7f693c0f09a0 msgr2=0x7f693c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:11.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.041+0000 7f693bfff700 1 -- 172.21.15.174:0/1034882718 shutdown_connections 2023-12-25T21:15:11.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.041+0000 7f693bfff700 1 -- 172.21.15.174:0/1034882718 wait complete. 2023-12-25T21:15:11.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.042+0000 7f693bfff700 1 Processor -- start 2023-12-25T21:15:11.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.042+0000 7f693bfff700 1 -- start start 2023-12-25T21:15:11.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.042+0000 7f693bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f693c0f51a0 0x7f693c1053e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:11.042 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.042+0000 7f693bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693c1058d0 con 0x7f693c0f51a0 2023-12-25T21:15:11.044 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.042+0000 7f693affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f693c0f51a0 0x7f693c1053e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:11.044 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.042+0000 7f693affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f693c0f51a0 0x7f693c1053e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:54264/0 (socket says 172.21.15.174:54264) 2023-12-25T21:15:11.044 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.042+0000 7f693affd700 1 -- 172.21.15.174:0/74796383 learned_addr learned my addr 172.21.15.174:0/74796383 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:15:11.044 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.043+0000 7f693affd700 1 -- 172.21.15.174:0/74796383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692c005040 con 0x7f693c0f51a0 2023-12-25T21:15:11.044 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.043+0000 7f693affd700 1 --2- 172.21.15.174:0/74796383 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f693c0f51a0 0x7f693c1053e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f692c013510 tx=0x7f692c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:11.044 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.043+0000 7f69237fe700 1 -- 172.21.15.174:0/74796383 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f692c009320 con 0x7f693c0f51a0 2023-12-25T21:15:11.044 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.043+0000 7f693bfff700 1 -- 172.21.15.174:0/74796383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f693c105ad0 con 0x7f693c0f51a0 2023-12-25T21:15:11.044 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.043+0000 7f69237fe700 1 -- 172.21.15.174:0/74796383 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f692c013f00 con 0x7f693c0f51a0 2023-12-25T21:15:11.045 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.043+0000 7f69237fe700 1 -- 172.21.15.174:0/74796383 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f692c014dc0 con 0x7f693c0f51a0 2023-12-25T21:15:11.045 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.043+0000 7f693bfff700 1 -- 172.21.15.174:0/74796383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f693c105f60 con 0x7f693c0f51a0 2023-12-25T21:15:11.045 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.044+0000 7f69237fe700 1 -- 172.21.15.174:0/74796383 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f692c0095f0 con 0x7f693c0f51a0 2023-12-25T21:15:11.045 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.044+0000 7f693bfff700 1 -- 172.21.15.174:0/74796383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693c045ad0 con 0x7f693c0f51a0 2023-12-25T21:15:11.045 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.044+0000 7f69237fe700 1 --2- 172.21.15.174:0/74796383 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f692402ff60 0x7f6924032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:11.045 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.044+0000 7f69237fe700 1 -- 172.21.15.174:0/74796383 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f692c041040 con 0x7f693c0f51a0 2023-12-25T21:15:11.045 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.044+0000 7f693a7fc700 1 --2- 172.21.15.174:0/74796383 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f692402ff60 0x7f6924032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:11.045 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.045+0000 7f693a7fc700 1 --2- 172.21.15.174:0/74796383 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f692402ff60 0x7f6924032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f69300096a0 tx=0x7f6930006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:11.048 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.049+0000 7f69237fe700 1 -- 172.21.15.174:0/74796383 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f692c014420 con 0x7f693c0f51a0 2023-12-25T21:15:11.257 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.257+0000 7f693bfff700 1 -- 172.21.15.174:0/74796383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f693c106e10 con 0x7f693c0f51a0 2023-12-25T21:15:11.258 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.258+0000 7f69237fe700 1 -- 172.21.15.174:0/74796383 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f692c010230 con 0x7f693c0f51a0 2023-12-25T21:15:11.258 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:11.258 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:11.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.260+0000 7f693bfff700 1 -- 172.21.15.174:0/74796383 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f692402ff60 msgr2=0x7f6924032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:11.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.260+0000 7f693bfff700 1 --2- 172.21.15.174:0/74796383 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f692402ff60 0x7f6924032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f69300096a0 tx=0x7f6930006b40).stop 2023-12-25T21:15:11.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.260+0000 7f693bfff700 1 -- 172.21.15.174:0/74796383 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f693c0f51a0 msgr2=0x7f693c1053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:11.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.260+0000 7f693bfff700 1 --2- 172.21.15.174:0/74796383 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f693c0f51a0 0x7f693c1053e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f692c013510 tx=0x7f692c00b040).stop 2023-12-25T21:15:11.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.260+0000 7f693bfff700 1 -- 172.21.15.174:0/74796383 shutdown_connections 2023-12-25T21:15:11.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.260+0000 7f693bfff700 1 --2- 172.21.15.174:0/74796383 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f692402ff60 0x7f6924032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:11.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.260+0000 7f693bfff700 1 --2- 172.21.15.174:0/74796383 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f693c0f51a0 0x7f693c1053e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:11.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.260+0000 7f693bfff700 1 -- 172.21.15.174:0/74796383 >> 172.21.15.174:0/74796383 conn(0x7f693c0f09a0 msgr2=0x7f693c0f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:11.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.260+0000 7f693bfff700 1 -- 172.21.15.174:0/74796383 shutdown_connections 2023-12-25T21:15:11.260 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:11.260+0000 7f693bfff700 1 -- 172.21.15.174:0/74796383 wait complete. 2023-12-25T21:15:11.261 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:11.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:11 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:10.646038+0000 mgr.smithi061.jxxeke (mgr.14164) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:11.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:11 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:11.258650+0000 mon.smithi061 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.174:0/74796383' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:15:12.680 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:15:12.680 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:15:13.888 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.886+0000 7fd4c3561700 1 -- 172.21.15.174:0/2462781652 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4bc0f6510 msgr2=0x7fd4bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:13.888 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.886+0000 7fd4c3561700 1 --2- 172.21.15.174:0/2462781652 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4bc0f6510 0x7fd4bc0f6930 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fd4ac009fb0 tx=0x7fd4ac009b30).stop 2023-12-25T21:15:13.888 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.886+0000 7fd4c3561700 1 -- 172.21.15.174:0/2462781652 shutdown_connections 2023-12-25T21:15:13.888 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.886+0000 7fd4c3561700 1 --2- 172.21.15.174:0/2462781652 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4bc0f6510 0x7fd4bc0f6930 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:13.888 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.886+0000 7fd4c3561700 1 -- 172.21.15.174:0/2462781652 >> 172.21.15.174:0/2462781652 conn(0x7fd4bc0f1c70 msgr2=0x7fd4bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:13.888 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.886+0000 7fd4c3561700 1 -- 172.21.15.174:0/2462781652 shutdown_connections 2023-12-25T21:15:13.889 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.886+0000 7fd4c3561700 1 -- 172.21.15.174:0/2462781652 wait complete. 2023-12-25T21:15:13.889 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.887+0000 7fd4c3561700 1 Processor -- start 2023-12-25T21:15:13.889 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.887+0000 7fd4c3561700 1 -- start start 2023-12-25T21:15:13.889 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.887+0000 7fd4c3561700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4bc0f6510 0x7fd4bc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:13.889 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.887+0000 7fd4c3561700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4bc106c20 con 0x7fd4bc0f6510 2023-12-25T21:15:13.889 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.887+0000 7fd4c12fd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4bc0f6510 0x7fd4bc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:13.889 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.887+0000 7fd4c12fd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4bc0f6510 0x7fd4bc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:54284/0 (socket says 172.21.15.174:54284) 2023-12-25T21:15:13.889 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.887+0000 7fd4c12fd700 1 -- 172.21.15.174:0/1486512812 learned_addr learned my addr 172.21.15.174:0/1486512812 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:15:13.889 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.887+0000 7fd4c12fd700 1 -- 172.21.15.174:0/1486512812 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac005040 con 0x7fd4bc0f6510 2023-12-25T21:15:13.889 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.888+0000 7fd4c12fd700 1 --2- 172.21.15.174:0/1486512812 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4bc0f6510 0x7fd4bc106730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fd4ac013510 tx=0x7fd4ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:13.889 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.888+0000 7fd4b1ffb700 1 -- 172.21.15.174:0/1486512812 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4ac009320 con 0x7fd4bc0f6510 2023-12-25T21:15:13.889 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.888+0000 7fd4c3561700 1 -- 172.21.15.174:0/1486512812 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4bc106e20 con 0x7fd4bc0f6510 2023-12-25T21:15:13.889 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.888+0000 7fd4b1ffb700 1 -- 172.21.15.174:0/1486512812 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4ac013f00 con 0x7fd4bc0f6510 2023-12-25T21:15:13.890 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.888+0000 7fd4b1ffb700 1 -- 172.21.15.174:0/1486512812 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4ac014d90 con 0x7fd4bc0f6510 2023-12-25T21:15:13.890 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.888+0000 7fd4c3561700 1 -- 172.21.15.174:0/1486512812 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4bc1072b0 con 0x7fd4bc0f6510 2023-12-25T21:15:13.890 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.889+0000 7fd4b1ffb700 1 -- 172.21.15.174:0/1486512812 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd4ac0263a0 con 0x7fd4bc0f6510 2023-12-25T21:15:13.891 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.889+0000 7fd4b1ffb700 1 --2- 172.21.15.174:0/1486512812 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd4a802ff60 0x7fd4a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:13.891 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.889+0000 7fd4b1ffb700 1 -- 172.21.15.174:0/1486512812 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd4ac0410b0 con 0x7fd4bc0f6510 2023-12-25T21:15:13.891 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.889+0000 7fd4c3561700 1 -- 172.21.15.174:0/1486512812 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4bc045ad0 con 0x7fd4bc0f6510 2023-12-25T21:15:13.891 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.889+0000 7fd4c0afc700 1 --2- 172.21.15.174:0/1486512812 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd4a802ff60 0x7fd4a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:13.891 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.890+0000 7fd4c0afc700 1 --2- 172.21.15.174:0/1486512812 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd4a802ff60 0x7fd4a8032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fd4b80096a0 tx=0x7fd4b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:13.893 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:13.894+0000 7fd4b1ffb700 1 -- 172.21.15.174:0/1486512812 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd4ac012920 con 0x7fd4bc0f6510 2023-12-25T21:15:13.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:13 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:12.646286+0000 mgr.smithi061.jxxeke (mgr.14164) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:14.098 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:14.098+0000 7fd4c3561700 1 -- 172.21.15.174:0/1486512812 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4bc0fc050 con 0x7fd4bc0f6510 2023-12-25T21:15:14.098 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:14.099+0000 7fd4b1ffb700 1 -- 172.21.15.174:0/1486512812 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd4ac00f340 con 0x7fd4bc0f6510 2023-12-25T21:15:14.099 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:14.099 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:14.100 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:14.100+0000 7fd4c3561700 1 -- 172.21.15.174:0/1486512812 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd4a802ff60 msgr2=0x7fd4a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:14.100 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:14.100+0000 7fd4c3561700 1 --2- 172.21.15.174:0/1486512812 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd4a802ff60 0x7fd4a8032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fd4b80096a0 tx=0x7fd4b8006b40).stop 2023-12-25T21:15:14.100 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:14.100+0000 7fd4c3561700 1 -- 172.21.15.174:0/1486512812 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4bc0f6510 msgr2=0x7fd4bc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:14.100 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:14.100+0000 7fd4c3561700 1 --2- 172.21.15.174:0/1486512812 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4bc0f6510 0x7fd4bc106730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fd4ac013510 tx=0x7fd4ac00b040).stop 2023-12-25T21:15:14.100 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:14.100+0000 7fd4c3561700 1 -- 172.21.15.174:0/1486512812 shutdown_connections 2023-12-25T21:15:14.101 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:14.100+0000 7fd4c3561700 1 --2- 172.21.15.174:0/1486512812 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd4a802ff60 0x7fd4a8032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:14.101 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:14.100+0000 7fd4c3561700 1 --2- 172.21.15.174:0/1486512812 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4bc0f6510 0x7fd4bc106730 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:14.101 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:14.100+0000 7fd4c3561700 1 -- 172.21.15.174:0/1486512812 >> 172.21.15.174:0/1486512812 conn(0x7fd4bc0f1c70 msgr2=0x7fd4bc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:14.101 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:14.100+0000 7fd4c3561700 1 -- 172.21.15.174:0/1486512812 shutdown_connections 2023-12-25T21:15:14.101 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:14.100+0000 7fd4c3561700 1 -- 172.21.15.174:0/1486512812 wait complete. 2023-12-25T21:15:14.101 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:14.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:14 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:14.099256+0000 mon.smithi061 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.174:0/1486512812' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:15:15.514 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:15:15.514 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:15:15.955 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:15 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:14.646642+0000 mgr.smithi061.jxxeke (mgr.14164) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:16.704 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.704+0000 7fb038579700 1 -- 172.21.15.174:0/3061749954 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0300f6510 msgr2=0x7fb0300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:16.704 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.704+0000 7fb038579700 1 --2- 172.21.15.174:0/3061749954 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0300f6510 0x7fb0300f6930 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fb02c009fb0 tx=0x7fb02c009b30).stop 2023-12-25T21:15:16.704 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.704+0000 7fb038579700 1 -- 172.21.15.174:0/3061749954 shutdown_connections 2023-12-25T21:15:16.705 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.704+0000 7fb038579700 1 --2- 172.21.15.174:0/3061749954 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0300f6510 0x7fb0300f6930 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:16.705 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.704+0000 7fb038579700 1 -- 172.21.15.174:0/3061749954 >> 172.21.15.174:0/3061749954 conn(0x7fb0300f1c70 msgr2=0x7fb0300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:16.705 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.704+0000 7fb038579700 1 -- 172.21.15.174:0/3061749954 shutdown_connections 2023-12-25T21:15:16.705 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.704+0000 7fb038579700 1 -- 172.21.15.174:0/3061749954 wait complete. 2023-12-25T21:15:16.705 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.705+0000 7fb038579700 1 Processor -- start 2023-12-25T21:15:16.705 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.705+0000 7fb038579700 1 -- start start 2023-12-25T21:15:16.707 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.705+0000 7fb038579700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0300f6510 0x7fb030106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:16.707 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.705+0000 7fb038579700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb030106c20 con 0x7fb0300f6510 2023-12-25T21:15:16.707 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.705+0000 7fb036315700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0300f6510 0x7fb030106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:16.707 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.705+0000 7fb036315700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0300f6510 0x7fb030106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:60296/0 (socket says 172.21.15.174:60296) 2023-12-25T21:15:16.707 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.705+0000 7fb036315700 1 -- 172.21.15.174:0/3018361926 learned_addr learned my addr 172.21.15.174:0/3018361926 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:15:16.707 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.705+0000 7fb036315700 1 -- 172.21.15.174:0/3018361926 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02c005040 con 0x7fb0300f6510 2023-12-25T21:15:16.707 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.705+0000 7fb036315700 1 --2- 172.21.15.174:0/3018361926 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0300f6510 0x7fb030106730 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fb02c013510 tx=0x7fb02c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:16.707 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.706+0000 7fb026ffd700 1 -- 172.21.15.174:0/3018361926 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb02c009320 con 0x7fb0300f6510 2023-12-25T21:15:16.707 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.706+0000 7fb026ffd700 1 -- 172.21.15.174:0/3018361926 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb02c013f00 con 0x7fb0300f6510 2023-12-25T21:15:16.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.706+0000 7fb038579700 1 -- 172.21.15.174:0/3018361926 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb030106e20 con 0x7fb0300f6510 2023-12-25T21:15:16.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.706+0000 7fb026ffd700 1 -- 172.21.15.174:0/3018361926 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb02c014d90 con 0x7fb0300f6510 2023-12-25T21:15:16.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.706+0000 7fb038579700 1 -- 172.21.15.174:0/3018361926 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0301072b0 con 0x7fb0300f6510 2023-12-25T21:15:16.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.706+0000 7fb038579700 1 -- 172.21.15.174:0/3018361926 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb030045ad0 con 0x7fb0300f6510 2023-12-25T21:15:16.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.706+0000 7fb026ffd700 1 -- 172.21.15.174:0/3018361926 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb02c0095f0 con 0x7fb0300f6510 2023-12-25T21:15:16.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.707+0000 7fb026ffd700 1 --2- 172.21.15.174:0/3018361926 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb01c02ff60 0x7fb01c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:16.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.707+0000 7fb026ffd700 1 -- 172.21.15.174:0/3018361926 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb02c041080 con 0x7fb0300f6510 2023-12-25T21:15:16.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.707+0000 7fb035b14700 1 --2- 172.21.15.174:0/3018361926 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb01c02ff60 0x7fb01c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:16.708 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.707+0000 7fb035b14700 1 --2- 172.21.15.174:0/3018361926 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb01c02ff60 0x7fb01c032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fb0200096a0 tx=0x7fb020006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:16.711 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.711+0000 7fb026ffd700 1 -- 172.21.15.174:0/3018361926 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb02c014420 con 0x7fb0300f6510 2023-12-25T21:15:16.916 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.916+0000 7fb038579700 1 -- 172.21.15.174:0/3018361926 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0300fbf60 con 0x7fb0300f6510 2023-12-25T21:15:16.917 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.917+0000 7fb026ffd700 1 -- 172.21.15.174:0/3018361926 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb02c010230 con 0x7fb0300f6510 2023-12-25T21:15:16.917 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:16.917 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:16.919 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.919+0000 7fb038579700 1 -- 172.21.15.174:0/3018361926 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb01c02ff60 msgr2=0x7fb01c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:16.919 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.919+0000 7fb038579700 1 --2- 172.21.15.174:0/3018361926 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb01c02ff60 0x7fb01c032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fb0200096a0 tx=0x7fb020006b40).stop 2023-12-25T21:15:16.919 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.919+0000 7fb038579700 1 -- 172.21.15.174:0/3018361926 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0300f6510 msgr2=0x7fb030106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:16.919 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.919+0000 7fb038579700 1 --2- 172.21.15.174:0/3018361926 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0300f6510 0x7fb030106730 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fb02c013510 tx=0x7fb02c00b040).stop 2023-12-25T21:15:16.919 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.919+0000 7fb038579700 1 -- 172.21.15.174:0/3018361926 shutdown_connections 2023-12-25T21:15:16.919 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.919+0000 7fb038579700 1 --2- 172.21.15.174:0/3018361926 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fb01c02ff60 0x7fb01c032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:16.919 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.919+0000 7fb038579700 1 --2- 172.21.15.174:0/3018361926 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0300f6510 0x7fb030106730 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:16.919 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.919+0000 7fb038579700 1 -- 172.21.15.174:0/3018361926 >> 172.21.15.174:0/3018361926 conn(0x7fb0300f1c70 msgr2=0x7fb0300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:16.919 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.919+0000 7fb038579700 1 -- 172.21.15.174:0/3018361926 shutdown_connections 2023-12-25T21:15:16.920 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:16.919+0000 7fb038579700 1 -- 172.21.15.174:0/3018361926 wait complete. 2023-12-25T21:15:16.920 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:17.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:17 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:16.917516+0000 mon.smithi061 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.174:0/3018361926' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:15:18.294 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:15:18.294 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:15:18.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:18 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:16.647048+0000 mgr.smithi061.jxxeke (mgr.14164) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:19.462 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.460+0000 7fe50e2fa700 1 -- 172.21.15.174:0/3693472162 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5080f51a0 msgr2=0x7fe5080f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:19.463 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.460+0000 7fe50e2fa700 1 --2- 172.21.15.174:0/3693472162 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5080f51a0 0x7fe5080f55c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fe4f8009fb0 tx=0x7fe4f8009b30).stop 2023-12-25T21:15:19.463 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.460+0000 7fe50e2fa700 1 -- 172.21.15.174:0/3693472162 shutdown_connections 2023-12-25T21:15:19.463 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.460+0000 7fe50e2fa700 1 --2- 172.21.15.174:0/3693472162 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5080f51a0 0x7fe5080f55c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:19.463 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.460+0000 7fe50e2fa700 1 -- 172.21.15.174:0/3693472162 >> 172.21.15.174:0/3693472162 conn(0x7fe5080f09a0 msgr2=0x7fe5080f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:19.464 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.460+0000 7fe50e2fa700 1 -- 172.21.15.174:0/3693472162 shutdown_connections 2023-12-25T21:15:19.464 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.460+0000 7fe50e2fa700 1 -- 172.21.15.174:0/3693472162 wait complete. 2023-12-25T21:15:19.464 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.460+0000 7fe50e2fa700 1 Processor -- start 2023-12-25T21:15:19.464 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.460+0000 7fe50e2fa700 1 -- start start 2023-12-25T21:15:19.464 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.460+0000 7fe50e2fa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5080f51a0 0x7fe5081091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:19.464 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.460+0000 7fe50e2fa700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe508060220 con 0x7fe5080f51a0 2023-12-25T21:15:19.464 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.461+0000 7fe50d2f8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5080f51a0 0x7fe5081091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:19.464 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.461+0000 7fe50d2f8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5080f51a0 0x7fe5081091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:60314/0 (socket says 172.21.15.174:60314) 2023-12-25T21:15:19.464 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.461+0000 7fe50d2f8700 1 -- 172.21.15.174:0/3596471216 learned_addr learned my addr 172.21.15.174:0/3596471216 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:15:19.464 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.461+0000 7fe50d2f8700 1 -- 172.21.15.174:0/3596471216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f8005040 con 0x7fe5080f51a0 2023-12-25T21:15:19.464 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.461+0000 7fe50d2f8700 1 --2- 172.21.15.174:0/3596471216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5080f51a0 0x7fe5081091d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fe4f8013510 tx=0x7fe4f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:19.465 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.461+0000 7fe4fdffb700 1 -- 172.21.15.174:0/3596471216 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4f8009320 con 0x7fe5080f51a0 2023-12-25T21:15:19.465 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.461+0000 7fe50e2fa700 1 -- 172.21.15.174:0/3596471216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe508060420 con 0x7fe5080f51a0 2023-12-25T21:15:19.465 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.461+0000 7fe4fdffb700 1 -- 172.21.15.174:0/3596471216 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4f8013f00 con 0x7fe5080f51a0 2023-12-25T21:15:19.465 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.461+0000 7fe4fdffb700 1 -- 172.21.15.174:0/3596471216 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4f8014dc0 con 0x7fe5080f51a0 2023-12-25T21:15:19.465 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.462+0000 7fe50e2fa700 1 -- 172.21.15.174:0/3596471216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe508109880 con 0x7fe5080f51a0 2023-12-25T21:15:19.466 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.462+0000 7fe4fdffb700 1 -- 172.21.15.174:0/3596471216 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe4f80095f0 con 0x7fe5080f51a0 2023-12-25T21:15:19.466 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.462+0000 7fe50e2fa700 1 -- 172.21.15.174:0/3596471216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe508059100 con 0x7fe5080f51a0 2023-12-25T21:15:19.466 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.462+0000 7fe4fdffb700 1 --2- 172.21.15.174:0/3596471216 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe4f402ffb0 0x7fe4f4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:19.466 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.462+0000 7fe4fdffb700 1 -- 172.21.15.174:0/3596471216 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe4f80410b0 con 0x7fe5080f51a0 2023-12-25T21:15:19.466 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.463+0000 7fe50caf7700 1 --2- 172.21.15.174:0/3596471216 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe4f402ffb0 0x7fe4f4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:19.466 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.463+0000 7fe50caf7700 1 --2- 172.21.15.174:0/3596471216 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe4f402ffb0 0x7fe4f4032470 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fe5040096a0 tx=0x7fe504006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:19.468 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.467+0000 7fe4fdffb700 1 -- 172.21.15.174:0/3596471216 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe4f806d2b0 con 0x7fe5080f51a0 2023-12-25T21:15:19.674 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.673+0000 7fe50e2fa700 1 -- 172.21.15.174:0/3596471216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe508045ad0 con 0x7fe5080f51a0 2023-12-25T21:15:19.674 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.673+0000 7fe4fdffb700 1 -- 172.21.15.174:0/3596471216 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe4f8014420 con 0x7fe5080f51a0 2023-12-25T21:15:19.674 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:19.674 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:19.676 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.675+0000 7fe50e2fa700 1 -- 172.21.15.174:0/3596471216 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe4f402ffb0 msgr2=0x7fe4f4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:19.676 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.675+0000 7fe50e2fa700 1 --2- 172.21.15.174:0/3596471216 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe4f402ffb0 0x7fe4f4032470 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fe5040096a0 tx=0x7fe504006b40).stop 2023-12-25T21:15:19.676 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.675+0000 7fe50e2fa700 1 -- 172.21.15.174:0/3596471216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5080f51a0 msgr2=0x7fe5081091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:19.676 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.675+0000 7fe50e2fa700 1 --2- 172.21.15.174:0/3596471216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5080f51a0 0x7fe5081091d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fe4f8013510 tx=0x7fe4f800b040).stop 2023-12-25T21:15:19.676 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.675+0000 7fe50e2fa700 1 -- 172.21.15.174:0/3596471216 shutdown_connections 2023-12-25T21:15:19.676 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.675+0000 7fe50e2fa700 1 --2- 172.21.15.174:0/3596471216 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fe4f402ffb0 0x7fe4f4032470 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:19.676 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.675+0000 7fe50e2fa700 1 --2- 172.21.15.174:0/3596471216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5080f51a0 0x7fe5081091d0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:19.676 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.675+0000 7fe50e2fa700 1 -- 172.21.15.174:0/3596471216 >> 172.21.15.174:0/3596471216 conn(0x7fe5080f09a0 msgr2=0x7fe5080f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:19.676 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.675+0000 7fe50e2fa700 1 -- 172.21.15.174:0/3596471216 shutdown_connections 2023-12-25T21:15:19.676 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:19.675+0000 7fe50e2fa700 1 -- 172.21.15.174:0/3596471216 wait complete. 2023-12-25T21:15:19.677 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:20.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:20 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:18.647484+0000 mgr.smithi061.jxxeke (mgr.14164) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:20.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:20 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:19.674749+0000 mon.smithi061 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.174:0/3596471216' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:15:21.054 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:15:21.055 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:15:22.195 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.194+0000 7fc80803a700 1 -- 172.21.15.174:0/4017646826 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8000f6510 msgr2=0x7fc8000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:22.196 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.194+0000 7fc80803a700 1 --2- 172.21.15.174:0/4017646826 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8000f6510 0x7fc8000f6930 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fc7fc009fb0 tx=0x7fc7fc009b30).stop 2023-12-25T21:15:22.196 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.194+0000 7fc80803a700 1 -- 172.21.15.174:0/4017646826 shutdown_connections 2023-12-25T21:15:22.196 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.194+0000 7fc80803a700 1 --2- 172.21.15.174:0/4017646826 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8000f6510 0x7fc8000f6930 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:22.196 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.194+0000 7fc80803a700 1 -- 172.21.15.174:0/4017646826 >> 172.21.15.174:0/4017646826 conn(0x7fc8000f1c70 msgr2=0x7fc8000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:22.196 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.194+0000 7fc80803a700 1 -- 172.21.15.174:0/4017646826 shutdown_connections 2023-12-25T21:15:22.196 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.194+0000 7fc80803a700 1 -- 172.21.15.174:0/4017646826 wait complete. 2023-12-25T21:15:22.196 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.195+0000 7fc80803a700 1 Processor -- start 2023-12-25T21:15:22.196 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.195+0000 7fc80803a700 1 -- start start 2023-12-25T21:15:22.196 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.195+0000 7fc80803a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc800108a30 0x7fc800108e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:22.196 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.195+0000 7fc80803a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc800109340 con 0x7fc800108a30 2023-12-25T21:15:22.200 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.195+0000 7fc805dd6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc800108a30 0x7fc800108e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:22.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.195+0000 7fc805dd6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc800108a30 0x7fc800108e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:60340/0 (socket says 172.21.15.174:60340) 2023-12-25T21:15:22.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.195+0000 7fc805dd6700 1 -- 172.21.15.174:0/769483631 learned_addr learned my addr 172.21.15.174:0/769483631 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:15:22.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.196+0000 7fc805dd6700 1 -- 172.21.15.174:0/769483631 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7fc005040 con 0x7fc800108a30 2023-12-25T21:15:22.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.196+0000 7fc805dd6700 1 --2- 172.21.15.174:0/769483631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc800108a30 0x7fc800108e50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fc7fc009f80 tx=0x7fc7fc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:22.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.196+0000 7fc7f67fc700 1 -- 172.21.15.174:0/769483631 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7fc0136c0 con 0x7fc800108a30 2023-12-25T21:15:22.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.196+0000 7fc80803a700 1 -- 172.21.15.174:0/769483631 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc800109540 con 0x7fc800108a30 2023-12-25T21:15:22.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.196+0000 7fc7f67fc700 1 -- 172.21.15.174:0/769483631 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7fc0095f0 con 0x7fc800108a30 2023-12-25T21:15:22.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.196+0000 7fc7f67fc700 1 -- 172.21.15.174:0/769483631 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7fc014420 con 0x7fc800108a30 2023-12-25T21:15:22.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.196+0000 7fc80803a700 1 -- 172.21.15.174:0/769483631 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc80010a090 con 0x7fc800108a30 2023-12-25T21:15:22.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.197+0000 7fc7f67fc700 1 -- 172.21.15.174:0/769483631 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7fc025060 con 0x7fc800108a30 2023-12-25T21:15:22.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.197+0000 7fc7f67fc700 1 --2- 172.21.15.174:0/769483631 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fc7ec02ff60 0x7fc7ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:22.201 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.197+0000 7fc7f67fc700 1 -- 172.21.15.174:0/769483631 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7fc040e70 con 0x7fc800108a30 2023-12-25T21:15:22.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.197+0000 7fc80803a700 1 -- 172.21.15.174:0/769483631 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc800059100 con 0x7fc800108a30 2023-12-25T21:15:22.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.197+0000 7fc8055d5700 1 --2- 172.21.15.174:0/769483631 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fc7ec02ff60 0x7fc7ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:22.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.197+0000 7fc8055d5700 1 --2- 172.21.15.174:0/769483631 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fc7ec02ff60 0x7fc7ec032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fc7f00096a0 tx=0x7fc7f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:22.202 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.202+0000 7fc7f67fc700 1 -- 172.21.15.174:0/769483631 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7fc028440 con 0x7fc800108a30 2023-12-25T21:15:22.410 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.409+0000 7fc80803a700 1 -- 172.21.15.174:0/769483631 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc80010a6f0 con 0x7fc800108a30 2023-12-25T21:15:22.411 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.410+0000 7fc7f67fc700 1 -- 172.21.15.174:0/769483631 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc7fc01d3a0 con 0x7fc800108a30 2023-12-25T21:15:22.411 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:22.411 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:22.413 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.412+0000 7fc80803a700 1 -- 172.21.15.174:0/769483631 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fc7ec02ff60 msgr2=0x7fc7ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:22.413 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.412+0000 7fc80803a700 1 --2- 172.21.15.174:0/769483631 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fc7ec02ff60 0x7fc7ec032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fc7f00096a0 tx=0x7fc7f0006b40).stop 2023-12-25T21:15:22.413 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.412+0000 7fc80803a700 1 -- 172.21.15.174:0/769483631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc800108a30 msgr2=0x7fc800108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:22.413 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.412+0000 7fc80803a700 1 --2- 172.21.15.174:0/769483631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc800108a30 0x7fc800108e50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fc7fc009f80 tx=0x7fc7fc009b30).stop 2023-12-25T21:15:22.413 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.412+0000 7fc80803a700 1 -- 172.21.15.174:0/769483631 shutdown_connections 2023-12-25T21:15:22.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.412+0000 7fc80803a700 1 --2- 172.21.15.174:0/769483631 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fc7ec02ff60 0x7fc7ec032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:22.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.412+0000 7fc80803a700 1 --2- 172.21.15.174:0/769483631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc800108a30 0x7fc800108e50 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:22.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.412+0000 7fc80803a700 1 -- 172.21.15.174:0/769483631 >> 172.21.15.174:0/769483631 conn(0x7fc8000f1c70 msgr2=0x7fc8000f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:22.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.412+0000 7fc80803a700 1 -- 172.21.15.174:0/769483631 shutdown_connections 2023-12-25T21:15:22.414 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:22.412+0000 7fc80803a700 1 -- 172.21.15.174:0/769483631 wait complete. 2023-12-25T21:15:22.414 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:22.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:22 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:20.647804+0000 mgr.smithi061.jxxeke (mgr.14164) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:23.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:23 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:22.411619+0000 mon.smithi061 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.174:0/769483631' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:15:23.850 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:15:23.850 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:15:24.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:24 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:22.648149+0000 mgr.smithi061.jxxeke (mgr.14164) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:25.029 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.029+0000 7efccbfff700 1 -- 172.21.15.174:0/3787036730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efccc0f78e0 msgr2=0x7efccc0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:25.030 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.029+0000 7efccbfff700 1 --2- 172.21.15.174:0/3787036730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efccc0f78e0 0x7efccc0f7d00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7efcbc009fb0 tx=0x7efcbc009b30).stop 2023-12-25T21:15:25.030 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.029+0000 7efccbfff700 1 -- 172.21.15.174:0/3787036730 shutdown_connections 2023-12-25T21:15:25.030 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.029+0000 7efccbfff700 1 --2- 172.21.15.174:0/3787036730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efccc0f78e0 0x7efccc0f7d00 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:25.030 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.029+0000 7efccbfff700 1 -- 172.21.15.174:0/3787036730 >> 172.21.15.174:0/3787036730 conn(0x7efccc0f0120 msgr2=0x7efccc0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:25.030 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.029+0000 7efccbfff700 1 -- 172.21.15.174:0/3787036730 shutdown_connections 2023-12-25T21:15:25.030 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.029+0000 7efccbfff700 1 -- 172.21.15.174:0/3787036730 wait complete. 2023-12-25T21:15:25.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.029+0000 7efccbfff700 1 Processor -- start 2023-12-25T21:15:25.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.030+0000 7efccbfff700 1 -- start start 2023-12-25T21:15:25.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.030+0000 7efccbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efccc0f78e0 0x7efccc1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:25.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.030+0000 7efccbfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efccc1069d0 con 0x7efccc0f78e0 2023-12-25T21:15:25.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.030+0000 7efccaffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efccc0f78e0 0x7efccc1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:25.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.030+0000 7efccaffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efccc0f78e0 0x7efccc1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:60368/0 (socket says 172.21.15.174:60368) 2023-12-25T21:15:25.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.030+0000 7efccaffd700 1 -- 172.21.15.174:0/4252064522 learned_addr learned my addr 172.21.15.174:0/4252064522 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:15:25.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.030+0000 7efccaffd700 1 -- 172.21.15.174:0/4252064522 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc005040 con 0x7efccc0f78e0 2023-12-25T21:15:25.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.030+0000 7efccaffd700 1 --2- 172.21.15.174:0/4252064522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efccc0f78e0 0x7efccc1064e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7efcbc013510 tx=0x7efcbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:25.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.031+0000 7efcb37fe700 1 -- 172.21.15.174:0/4252064522 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcbc009320 con 0x7efccc0f78e0 2023-12-25T21:15:25.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.031+0000 7efccbfff700 1 -- 172.21.15.174:0/4252064522 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efccc106bd0 con 0x7efccc0f78e0 2023-12-25T21:15:25.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.031+0000 7efcb37fe700 1 -- 172.21.15.174:0/4252064522 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efcbc013f00 con 0x7efccc0f78e0 2023-12-25T21:15:25.032 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.031+0000 7efcb37fe700 1 -- 172.21.15.174:0/4252064522 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcbc014dc0 con 0x7efccc0f78e0 2023-12-25T21:15:25.033 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.031+0000 7efccbfff700 1 -- 172.21.15.174:0/4252064522 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efccc107060 con 0x7efccc0f78e0 2023-12-25T21:15:25.033 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.031+0000 7efcb37fe700 1 -- 172.21.15.174:0/4252064522 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efcbc0095f0 con 0x7efccc0f78e0 2023-12-25T21:15:25.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.031+0000 7efccbfff700 1 -- 172.21.15.174:0/4252064522 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efccc045ad0 con 0x7efccc0f78e0 2023-12-25T21:15:25.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.032+0000 7efcb37fe700 1 --2- 172.21.15.174:0/4252064522 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efcb402ff60 0x7efcb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:25.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.032+0000 7efcb37fe700 1 -- 172.21.15.174:0/4252064522 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efcbc041100 con 0x7efccc0f78e0 2023-12-25T21:15:25.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.032+0000 7efcca7fc700 1 --2- 172.21.15.174:0/4252064522 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efcb402ff60 0x7efcb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:25.034 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.032+0000 7efcca7fc700 1 --2- 172.21.15.174:0/4252064522 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efcb402ff60 0x7efcb4032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7efcc00096a0 tx=0x7efcc0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:25.037 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.036+0000 7efcb37fe700 1 -- 172.21.15.174:0/4252064522 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efcbc014420 con 0x7efccc0f78e0 2023-12-25T21:15:25.243 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.242+0000 7efccbfff700 1 -- 172.21.15.174:0/4252064522 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efccc107f10 con 0x7efccc0f78e0 2023-12-25T21:15:25.244 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.243+0000 7efcb37fe700 1 -- 172.21.15.174:0/4252064522 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efcbc010230 con 0x7efccc0f78e0 2023-12-25T21:15:25.244 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:25.244 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:25.246 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.245+0000 7efccbfff700 1 -- 172.21.15.174:0/4252064522 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efcb402ff60 msgr2=0x7efcb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:25.246 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.245+0000 7efccbfff700 1 --2- 172.21.15.174:0/4252064522 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efcb402ff60 0x7efcb4032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7efcc00096a0 tx=0x7efcc0006b40).stop 2023-12-25T21:15:25.246 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.245+0000 7efccbfff700 1 -- 172.21.15.174:0/4252064522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efccc0f78e0 msgr2=0x7efccc1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:25.246 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.245+0000 7efccbfff700 1 --2- 172.21.15.174:0/4252064522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efccc0f78e0 0x7efccc1064e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7efcbc013510 tx=0x7efcbc00b040).stop 2023-12-25T21:15:25.246 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.245+0000 7efccbfff700 1 -- 172.21.15.174:0/4252064522 shutdown_connections 2023-12-25T21:15:25.246 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.245+0000 7efccbfff700 1 --2- 172.21.15.174:0/4252064522 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7efcb402ff60 0x7efcb4032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:25.246 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.245+0000 7efccbfff700 1 --2- 172.21.15.174:0/4252064522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efccc0f78e0 0x7efccc1064e0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:25.246 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.245+0000 7efccbfff700 1 -- 172.21.15.174:0/4252064522 >> 172.21.15.174:0/4252064522 conn(0x7efccc0f0120 msgr2=0x7efccc0f0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:25.246 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.245+0000 7efccbfff700 1 -- 172.21.15.174:0/4252064522 shutdown_connections 2023-12-25T21:15:25.246 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:25.245+0000 7efccbfff700 1 -- 172.21.15.174:0/4252064522 wait complete. 2023-12-25T21:15:25.247 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:25.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:25 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:25.244373+0000 mon.smithi061 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.174:0/4252064522' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:15:26.616 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:15:26.616 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:15:26.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:26 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:24.648537+0000 mgr.smithi061.jxxeke (mgr.14164) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:27.810 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.808+0000 7f1635904700 1 -- 172.21.15.174:0/2944668315 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16300f5410 msgr2=0x7f16300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:27.810 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.808+0000 7f1635904700 1 --2- 172.21.15.174:0/2944668315 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16300f5410 0x7f16300f5830 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f1620009fb0 tx=0x7f1620009b30).stop 2023-12-25T21:15:27.810 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.808+0000 7f1635904700 1 -- 172.21.15.174:0/2944668315 shutdown_connections 2023-12-25T21:15:27.811 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.808+0000 7f1635904700 1 --2- 172.21.15.174:0/2944668315 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16300f5410 0x7f16300f5830 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:27.811 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.808+0000 7f1635904700 1 -- 172.21.15.174:0/2944668315 >> 172.21.15.174:0/2944668315 conn(0x7f16300f0b70 msgr2=0x7f16300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:27.811 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.808+0000 7f1635904700 1 -- 172.21.15.174:0/2944668315 shutdown_connections 2023-12-25T21:15:27.811 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.809+0000 7f1635904700 1 -- 172.21.15.174:0/2944668315 wait complete. 2023-12-25T21:15:27.811 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.809+0000 7f1635904700 1 Processor -- start 2023-12-25T21:15:27.811 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.809+0000 7f1635904700 1 -- start start 2023-12-25T21:15:27.811 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.810+0000 7f1635904700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16300f5410 0x7f1630109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:27.811 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.810+0000 7f1635904700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1630060220 con 0x7f16300f5410 2023-12-25T21:15:27.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.810+0000 7f162effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16300f5410 0x7f1630109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:27.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.810+0000 7f162effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16300f5410 0x7f1630109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:59344/0 (socket says 172.21.15.174:59344) 2023-12-25T21:15:27.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.810+0000 7f162effd700 1 -- 172.21.15.174:0/1912676304 learned_addr learned my addr 172.21.15.174:0/1912676304 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:15:27.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.810+0000 7f162effd700 1 -- 172.21.15.174:0/1912676304 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1620005040 con 0x7f16300f5410 2023-12-25T21:15:27.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.810+0000 7f162effd700 1 --2- 172.21.15.174:0/1912676304 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16300f5410 0x7f1630109420 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f1620013510 tx=0x7f162000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:27.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.811+0000 7f16177fe700 1 -- 172.21.15.174:0/1912676304 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1620009320 con 0x7f16300f5410 2023-12-25T21:15:27.813 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.811+0000 7f1635904700 1 -- 172.21.15.174:0/1912676304 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1630060420 con 0x7f16300f5410 2023-12-25T21:15:27.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.811+0000 7f16177fe700 1 -- 172.21.15.174:0/1912676304 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1620013f00 con 0x7f16300f5410 2023-12-25T21:15:27.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.811+0000 7f16177fe700 1 -- 172.21.15.174:0/1912676304 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1620014d90 con 0x7f16300f5410 2023-12-25T21:15:27.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.811+0000 7f1635904700 1 -- 172.21.15.174:0/1912676304 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1630109ad0 con 0x7f16300f5410 2023-12-25T21:15:27.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.811+0000 7f16177fe700 1 -- 172.21.15.174:0/1912676304 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1620014420 con 0x7f16300f5410 2023-12-25T21:15:27.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.812+0000 7f1635904700 1 -- 172.21.15.174:0/1912676304 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1630059100 con 0x7f16300f5410 2023-12-25T21:15:27.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.812+0000 7f16177fe700 1 --2- 172.21.15.174:0/1912676304 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f161802ff60 0x7f1618032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:27.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.812+0000 7f16177fe700 1 -- 172.21.15.174:0/1912676304 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1620040ff0 con 0x7f16300f5410 2023-12-25T21:15:27.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.812+0000 7f162e7fc700 1 --2- 172.21.15.174:0/1912676304 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f161802ff60 0x7f1618032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:27.814 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.812+0000 7f162e7fc700 1 --2- 172.21.15.174:0/1912676304 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f161802ff60 0x7f1618032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f16240096a0 tx=0x7f1624006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:27.817 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:27.817+0000 7f16177fe700 1 -- 172.21.15.174:0/1912676304 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16200095f0 con 0x7f16300f5410 2023-12-25T21:15:28.019 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:28.019+0000 7f1635904700 1 -- 172.21.15.174:0/1912676304 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1630045ad0 con 0x7f16300f5410 2023-12-25T21:15:28.020 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:28.019+0000 7f16177fe700 1 -- 172.21.15.174:0/1912676304 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f162001db70 con 0x7f16300f5410 2023-12-25T21:15:28.020 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:28.020 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:28.022 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:28.021+0000 7f1635904700 1 -- 172.21.15.174:0/1912676304 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f161802ff60 msgr2=0x7f1618032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:28.022 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:28.021+0000 7f1635904700 1 --2- 172.21.15.174:0/1912676304 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f161802ff60 0x7f1618032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f16240096a0 tx=0x7f1624006b40).stop 2023-12-25T21:15:28.022 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:28.021+0000 7f1635904700 1 -- 172.21.15.174:0/1912676304 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16300f5410 msgr2=0x7f1630109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:28.022 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:28.021+0000 7f1635904700 1 --2- 172.21.15.174:0/1912676304 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16300f5410 0x7f1630109420 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f1620013510 tx=0x7f162000b040).stop 2023-12-25T21:15:28.022 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:28.021+0000 7f1635904700 1 -- 172.21.15.174:0/1912676304 shutdown_connections 2023-12-25T21:15:28.022 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:28.021+0000 7f1635904700 1 --2- 172.21.15.174:0/1912676304 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f161802ff60 0x7f1618032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:28.022 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:28.021+0000 7f1635904700 1 --2- 172.21.15.174:0/1912676304 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16300f5410 0x7f1630109420 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:28.022 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:28.021+0000 7f1635904700 1 -- 172.21.15.174:0/1912676304 >> 172.21.15.174:0/1912676304 conn(0x7f16300f0b70 msgr2=0x7f16300f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:28.022 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:28.021+0000 7f1635904700 1 -- 172.21.15.174:0/1912676304 shutdown_connections 2023-12-25T21:15:28.022 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:28.021+0000 7f1635904700 1 -- 172.21.15.174:0/1912676304 wait complete. 2023-12-25T21:15:28.023 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:28.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:28 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:26.648904+0000 mgr.smithi061.jxxeke (mgr.14164) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:28.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:28 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:28.020948+0000 mon.smithi061 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.174:0/1912676304' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:15:29.432 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:15:29.433 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:15:30.616 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.609+0000 7fd6d1cf9700 1 -- 172.21.15.174:0/1058209089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6cc0f51a0 msgr2=0x7fd6cc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:30.616 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.609+0000 7fd6d1cf9700 1 --2- 172.21.15.174:0/1058209089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6cc0f51a0 0x7fd6cc0f55c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fd6c0009fb0 tx=0x7fd6c0009b30).stop 2023-12-25T21:15:30.616 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.609+0000 7fd6d1cf9700 1 -- 172.21.15.174:0/1058209089 shutdown_connections 2023-12-25T21:15:30.616 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.609+0000 7fd6d1cf9700 1 --2- 172.21.15.174:0/1058209089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6cc0f51a0 0x7fd6cc0f55c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:30.616 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.609+0000 7fd6d1cf9700 1 -- 172.21.15.174:0/1058209089 >> 172.21.15.174:0/1058209089 conn(0x7fd6cc0f09a0 msgr2=0x7fd6cc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:30.616 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.609+0000 7fd6d1cf9700 1 -- 172.21.15.174:0/1058209089 shutdown_connections 2023-12-25T21:15:30.617 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.609+0000 7fd6d1cf9700 1 -- 172.21.15.174:0/1058209089 wait complete. 2023-12-25T21:15:30.617 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.609+0000 7fd6d1cf9700 1 Processor -- start 2023-12-25T21:15:30.617 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.609+0000 7fd6d1cf9700 1 -- start start 2023-12-25T21:15:30.617 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.610+0000 7fd6d1cf9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6cc0f51a0 0x7fd6cc1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:30.617 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.610+0000 7fd6d1cf9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc1058a0 con 0x7fd6cc0f51a0 2023-12-25T21:15:30.617 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.610+0000 7fd6d0cf7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6cc0f51a0 0x7fd6cc1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:30.617 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.610+0000 7fd6d0cf7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6cc0f51a0 0x7fd6cc1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:59352/0 (socket says 172.21.15.174:59352) 2023-12-25T21:15:30.617 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.610+0000 7fd6d0cf7700 1 -- 172.21.15.174:0/187914205 learned_addr learned my addr 172.21.15.174:0/187914205 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:15:30.617 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.610+0000 7fd6d0cf7700 1 -- 172.21.15.174:0/187914205 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c0005040 con 0x7fd6cc0f51a0 2023-12-25T21:15:30.617 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.610+0000 7fd6d0cf7700 1 --2- 172.21.15.174:0/187914205 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6cc0f51a0 0x7fd6cc1053b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fd6c0013510 tx=0x7fd6c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:30.617 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.610+0000 7fd6c97fa700 1 -- 172.21.15.174:0/187914205 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6c0009320 con 0x7fd6cc0f51a0 2023-12-25T21:15:30.617 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.610+0000 7fd6d1cf9700 1 -- 172.21.15.174:0/187914205 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6cc105aa0 con 0x7fd6cc0f51a0 2023-12-25T21:15:30.617 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.610+0000 7fd6c97fa700 1 -- 172.21.15.174:0/187914205 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd6c0013f00 con 0x7fd6cc0f51a0 2023-12-25T21:15:30.618 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.610+0000 7fd6c97fa700 1 -- 172.21.15.174:0/187914205 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6c0014dc0 con 0x7fd6cc0f51a0 2023-12-25T21:15:30.618 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.610+0000 7fd6d1cf9700 1 -- 172.21.15.174:0/187914205 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6cc105f30 con 0x7fd6cc0f51a0 2023-12-25T21:15:30.618 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.611+0000 7fd6c97fa700 1 -- 172.21.15.174:0/187914205 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd6c001d3d0 con 0x7fd6cc0f51a0 2023-12-25T21:15:30.618 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.611+0000 7fd6d1cf9700 1 -- 172.21.15.174:0/187914205 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6cc045ad0 con 0x7fd6cc0f51a0 2023-12-25T21:15:30.618 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.611+0000 7fd6c97fa700 1 --2- 172.21.15.174:0/187914205 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd6bc02ff60 0x7fd6bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:30.618 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.611+0000 7fd6c97fa700 1 -- 172.21.15.174:0/187914205 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd6c00413b0 con 0x7fd6cc0f51a0 2023-12-25T21:15:30.618 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.612+0000 7fd6cbfff700 1 --2- 172.21.15.174:0/187914205 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd6bc02ff60 0x7fd6bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:30.618 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.612+0000 7fd6cbfff700 1 --2- 172.21.15.174:0/187914205 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd6bc02ff60 0x7fd6bc032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fd6b80096a0 tx=0x7fd6b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:30.619 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.616+0000 7fd6c97fa700 1 -- 172.21.15.174:0/187914205 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd6c001cad0 con 0x7fd6cc0f51a0 2023-12-25T21:15:30.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:30 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:28.649267+0000 mgr.smithi061.jxxeke (mgr.14164) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:30.823 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.822+0000 7fd6d1cf9700 1 -- 172.21.15.174:0/187914205 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd6cc106de0 con 0x7fd6cc0f51a0 2023-12-25T21:15:30.973 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.823+0000 7fd6c97fa700 1 -- 172.21.15.174:0/187914205 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd6c001cad0 con 0x7fd6cc0f51a0 2023-12-25T21:15:30.974 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:30.974 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:30.974 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.825+0000 7fd6d1cf9700 1 -- 172.21.15.174:0/187914205 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd6bc02ff60 msgr2=0x7fd6bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:30.974 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.825+0000 7fd6d1cf9700 1 --2- 172.21.15.174:0/187914205 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd6bc02ff60 0x7fd6bc032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fd6b80096a0 tx=0x7fd6b8006b40).stop 2023-12-25T21:15:30.975 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.825+0000 7fd6d1cf9700 1 -- 172.21.15.174:0/187914205 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6cc0f51a0 msgr2=0x7fd6cc1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:30.975 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.825+0000 7fd6d1cf9700 1 --2- 172.21.15.174:0/187914205 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6cc0f51a0 0x7fd6cc1053b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fd6c0013510 tx=0x7fd6c000b040).stop 2023-12-25T21:15:30.975 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.825+0000 7fd6d1cf9700 1 -- 172.21.15.174:0/187914205 shutdown_connections 2023-12-25T21:15:30.975 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.825+0000 7fd6d1cf9700 1 --2- 172.21.15.174:0/187914205 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fd6bc02ff60 0x7fd6bc032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:30.975 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.825+0000 7fd6d1cf9700 1 --2- 172.21.15.174:0/187914205 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6cc0f51a0 0x7fd6cc1053b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:30.975 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.825+0000 7fd6d1cf9700 1 -- 172.21.15.174:0/187914205 >> 172.21.15.174:0/187914205 conn(0x7fd6cc0f09a0 msgr2=0x7fd6cc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:30.975 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.825+0000 7fd6d1cf9700 1 -- 172.21.15.174:0/187914205 shutdown_connections 2023-12-25T21:15:30.975 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:30.825+0000 7fd6d1cf9700 1 -- 172.21.15.174:0/187914205 wait complete. 2023-12-25T21:15:30.975 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:31.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:31 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:30.824679+0000 mon.smithi061 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.174:0/187914205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:15:32.223 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:15:32.224 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:15:32.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:32 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:30.649618+0000 mgr.smithi061.jxxeke (mgr.14164) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:33.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.450+0000 7f673f692700 1 -- 172.21.15.174:0/3525516393 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67380f5410 msgr2=0x7f67380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:33.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.450+0000 7f673f692700 1 --2- 172.21.15.174:0/3525516393 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67380f5410 0x7f67380f5830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f6728009fb0 tx=0x7f6728009b30).stop 2023-12-25T21:15:33.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.450+0000 7f673f692700 1 -- 172.21.15.174:0/3525516393 shutdown_connections 2023-12-25T21:15:33.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.450+0000 7f673f692700 1 --2- 172.21.15.174:0/3525516393 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67380f5410 0x7f67380f5830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:33.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.450+0000 7f673f692700 1 -- 172.21.15.174:0/3525516393 >> 172.21.15.174:0/3525516393 conn(0x7f67380f0b70 msgr2=0x7f67380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:33.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.450+0000 7f673f692700 1 -- 172.21.15.174:0/3525516393 shutdown_connections 2023-12-25T21:15:33.452 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.450+0000 7f673f692700 1 -- 172.21.15.174:0/3525516393 wait complete. 2023-12-25T21:15:33.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.450+0000 7f673f692700 1 Processor -- start 2023-12-25T21:15:33.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.450+0000 7f673f692700 1 -- start start 2023-12-25T21:15:33.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.450+0000 7f673f692700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67380f5410 0x7f6738105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:33.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.450+0000 7f673f692700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6738105b20 con 0x7f67380f5410 2023-12-25T21:15:33.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.451+0000 7f673d42e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67380f5410 0x7f6738105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:33.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.451+0000 7f673d42e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67380f5410 0x7f6738105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:59368/0 (socket says 172.21.15.174:59368) 2023-12-25T21:15:33.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.451+0000 7f673d42e700 1 -- 172.21.15.174:0/2081711635 learned_addr learned my addr 172.21.15.174:0/2081711635 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:15:33.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.451+0000 7f673d42e700 1 -- 172.21.15.174:0/2081711635 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6728005040 con 0x7f67380f5410 2023-12-25T21:15:33.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.451+0000 7f673d42e700 1 --2- 172.21.15.174:0/2081711635 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67380f5410 0x7f6738105630 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f6728014000 tx=0x7f672800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:33.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.451+0000 7f672dffb700 1 -- 172.21.15.174:0/2081711635 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67280095f0 con 0x7f67380f5410 2023-12-25T21:15:33.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.451+0000 7f673f692700 1 -- 172.21.15.174:0/2081711635 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6738105d20 con 0x7f67380f5410 2023-12-25T21:15:33.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.451+0000 7f672dffb700 1 -- 172.21.15.174:0/2081711635 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6728007e70 con 0x7f67380f5410 2023-12-25T21:15:33.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.451+0000 7f672dffb700 1 -- 172.21.15.174:0/2081711635 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6728014ea0 con 0x7f67380f5410 2023-12-25T21:15:33.454 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.451+0000 7f673f692700 1 -- 172.21.15.174:0/2081711635 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67381061b0 con 0x7f67380f5410 2023-12-25T21:15:33.454 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.452+0000 7f672dffb700 1 -- 172.21.15.174:0/2081711635 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6728025070 con 0x7f67380f5410 2023-12-25T21:15:33.454 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.452+0000 7f673f692700 1 -- 172.21.15.174:0/2081711635 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6738045ad0 con 0x7f67380f5410 2023-12-25T21:15:33.454 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.452+0000 7f672dffb700 1 --2- 172.21.15.174:0/2081711635 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f672402ff60 0x7f6724032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:33.455 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.453+0000 7f672dffb700 1 -- 172.21.15.174:0/2081711635 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6728040980 con 0x7f67380f5410 2023-12-25T21:15:33.455 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.453+0000 7f673cc2d700 1 --2- 172.21.15.174:0/2081711635 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f672402ff60 0x7f6724032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:33.455 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.453+0000 7f673cc2d700 1 --2- 172.21.15.174:0/2081711635 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f672402ff60 0x7f6724032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f67340096a0 tx=0x7f6734006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:33.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.457+0000 7f672dffb700 1 -- 172.21.15.174:0/2081711635 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f67280079c0 con 0x7f67380f5410 2023-12-25T21:15:33.662 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.661+0000 7f673f692700 1 -- 172.21.15.174:0/2081711635 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67380fae60 con 0x7f67380f5410 2023-12-25T21:15:33.662 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.661+0000 7f672dffb700 1 -- 172.21.15.174:0/2081711635 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f672800f300 con 0x7f67380f5410 2023-12-25T21:15:33.662 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:33.663 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:33.664 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.663+0000 7f673f692700 1 -- 172.21.15.174:0/2081711635 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f672402ff60 msgr2=0x7f6724032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:33.664 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.663+0000 7f673f692700 1 --2- 172.21.15.174:0/2081711635 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f672402ff60 0x7f6724032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f67340096a0 tx=0x7f6734006b40).stop 2023-12-25T21:15:33.664 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.663+0000 7f673f692700 1 -- 172.21.15.174:0/2081711635 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67380f5410 msgr2=0x7f6738105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:33.664 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.663+0000 7f673f692700 1 --2- 172.21.15.174:0/2081711635 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67380f5410 0x7f6738105630 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f6728014000 tx=0x7f672800b040).stop 2023-12-25T21:15:33.664 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.663+0000 7f673f692700 1 -- 172.21.15.174:0/2081711635 shutdown_connections 2023-12-25T21:15:33.664 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.663+0000 7f673f692700 1 --2- 172.21.15.174:0/2081711635 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f672402ff60 0x7f6724032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:33.664 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.663+0000 7f673f692700 1 --2- 172.21.15.174:0/2081711635 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67380f5410 0x7f6738105630 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:33.664 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.663+0000 7f673f692700 1 -- 172.21.15.174:0/2081711635 >> 172.21.15.174:0/2081711635 conn(0x7f67380f0b70 msgr2=0x7f67380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:33.665 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.663+0000 7f673f692700 1 -- 172.21.15.174:0/2081711635 shutdown_connections 2023-12-25T21:15:33.665 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:33.663+0000 7f673f692700 1 -- 172.21.15.174:0/2081711635 wait complete. 2023-12-25T21:15:33.665 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:34.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:34 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:32.649999+0000 mgr.smithi061.jxxeke (mgr.14164) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:34.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:34 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:33.663058+0000 mon.smithi061 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.174:0/2081711635' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:15:35.089 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:15:35.089 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:15:36.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.283+0000 7f891ad26700 1 -- 172.21.15.174:0/420577663 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89140f62a0 msgr2=0x7f89140f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:36.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.283+0000 7f891ad26700 1 --2- 172.21.15.174:0/420577663 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89140f62a0 0x7f89140f66c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f8910009fb0 tx=0x7f8910009b30).stop 2023-12-25T21:15:36.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.283+0000 7f891ad26700 1 -- 172.21.15.174:0/420577663 shutdown_connections 2023-12-25T21:15:36.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.283+0000 7f891ad26700 1 --2- 172.21.15.174:0/420577663 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89140f62a0 0x7f89140f66c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:36.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.283+0000 7f891ad26700 1 -- 172.21.15.174:0/420577663 >> 172.21.15.174:0/420577663 conn(0x7f89140f1aa0 msgr2=0x7f89140f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:36.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.283+0000 7f891ad26700 1 -- 172.21.15.174:0/420577663 shutdown_connections 2023-12-25T21:15:36.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.283+0000 7f891ad26700 1 -- 172.21.15.174:0/420577663 wait complete. 2023-12-25T21:15:36.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.283+0000 7f891ad26700 1 Processor -- start 2023-12-25T21:15:36.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.283+0000 7f891ad26700 1 -- start start 2023-12-25T21:15:36.286 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.283+0000 7f891ad26700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89140f62a0 0x7f89141064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:36.286 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.283+0000 7f891ad26700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89141069a0 con 0x7f89140f62a0 2023-12-25T21:15:36.286 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.284+0000 7f8919d24700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89140f62a0 0x7f89141064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:36.286 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.284+0000 7f8919d24700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89140f62a0 0x7f89141064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:59378/0 (socket says 172.21.15.174:59378) 2023-12-25T21:15:36.286 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.284+0000 7f8919d24700 1 -- 172.21.15.174:0/3442602337 learned_addr learned my addr 172.21.15.174:0/3442602337 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:15:36.286 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.284+0000 7f8919d24700 1 -- 172.21.15.174:0/3442602337 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8910005040 con 0x7f89140f62a0 2023-12-25T21:15:36.286 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.284+0000 7f8919d24700 1 --2- 172.21.15.174:0/3442602337 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89140f62a0 0x7f89141064b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f8910013510 tx=0x7f891000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:36.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.284+0000 7f89067fc700 1 -- 172.21.15.174:0/3442602337 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8910009320 con 0x7f89140f62a0 2023-12-25T21:15:36.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.285+0000 7f891ad26700 1 -- 172.21.15.174:0/3442602337 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8914106ba0 con 0x7f89140f62a0 2023-12-25T21:15:36.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.285+0000 7f891ad26700 1 -- 172.21.15.174:0/3442602337 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8914107030 con 0x7f89140f62a0 2023-12-25T21:15:36.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.285+0000 7f89067fc700 1 -- 172.21.15.174:0/3442602337 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8910013f00 con 0x7f89140f62a0 2023-12-25T21:15:36.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.285+0000 7f89067fc700 1 -- 172.21.15.174:0/3442602337 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8910014d90 con 0x7f89140f62a0 2023-12-25T21:15:36.287 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.285+0000 7f89067fc700 1 -- 172.21.15.174:0/3442602337 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f89100263a0 con 0x7f89140f62a0 2023-12-25T21:15:36.288 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.285+0000 7f891ad26700 1 -- 172.21.15.174:0/3442602337 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8914045ad0 con 0x7f89140f62a0 2023-12-25T21:15:36.288 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.285+0000 7f89067fc700 1 --2- 172.21.15.174:0/3442602337 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f890002ff60 0x7f8900032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:36.288 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.285+0000 7f89067fc700 1 -- 172.21.15.174:0/3442602337 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f89100414b0 con 0x7f89140f62a0 2023-12-25T21:15:36.288 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.286+0000 7f8919523700 1 --2- 172.21.15.174:0/3442602337 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f890002ff60 0x7f8900032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:36.288 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.286+0000 7f8919523700 1 --2- 172.21.15.174:0/3442602337 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f890002ff60 0x7f8900032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f89080096a0 tx=0x7f8908006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:36.291 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.290+0000 7f89067fc700 1 -- 172.21.15.174:0/3442602337 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8910012510 con 0x7f89140f62a0 2023-12-25T21:15:36.503 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.501+0000 7f891ad26700 1 -- 172.21.15.174:0/3442602337 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8914107ee0 con 0x7f89140f62a0 2023-12-25T21:15:36.503 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.502+0000 7f89067fc700 1 -- 172.21.15.174:0/3442602337 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8910012330 con 0x7f89140f62a0 2023-12-25T21:15:36.504 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:36.504 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:36.505 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.504+0000 7f891ad26700 1 -- 172.21.15.174:0/3442602337 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f890002ff60 msgr2=0x7f8900032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:36.505 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.504+0000 7f891ad26700 1 --2- 172.21.15.174:0/3442602337 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f890002ff60 0x7f8900032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f89080096a0 tx=0x7f8908006b40).stop 2023-12-25T21:15:36.505 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.504+0000 7f891ad26700 1 -- 172.21.15.174:0/3442602337 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89140f62a0 msgr2=0x7f89141064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:36.505 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.504+0000 7f891ad26700 1 --2- 172.21.15.174:0/3442602337 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89140f62a0 0x7f89141064b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f8910013510 tx=0x7f891000b040).stop 2023-12-25T21:15:36.505 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.504+0000 7f891ad26700 1 -- 172.21.15.174:0/3442602337 shutdown_connections 2023-12-25T21:15:36.505 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.504+0000 7f891ad26700 1 --2- 172.21.15.174:0/3442602337 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f890002ff60 0x7f8900032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:36.505 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.504+0000 7f891ad26700 1 --2- 172.21.15.174:0/3442602337 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89140f62a0 0x7f89141064b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:36.505 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.504+0000 7f891ad26700 1 -- 172.21.15.174:0/3442602337 >> 172.21.15.174:0/3442602337 conn(0x7f89140f1aa0 msgr2=0x7f89140f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:36.505 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.504+0000 7f891ad26700 1 -- 172.21.15.174:0/3442602337 shutdown_connections 2023-12-25T21:15:36.505 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:36.504+0000 7f891ad26700 1 -- 172.21.15.174:0/3442602337 wait complete. 2023-12-25T21:15:36.506 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:36.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:36 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:34.650386+0000 mgr.smithi061.jxxeke (mgr.14164) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:37.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:37 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:36.503720+0000 mon.smithi061 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.174:0/3442602337' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:15:37.894 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:15:37.894 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:15:38.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:38 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:36.650756+0000 mgr.smithi061.jxxeke (mgr.14164) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:39.146 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.144+0000 7f60e759e700 1 -- 172.21.15.174:0/1714909986 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60e80f51a0 msgr2=0x7f60e80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:39.146 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.144+0000 7f60e759e700 1 --2- 172.21.15.174:0/1714909986 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60e80f51a0 0x7f60e80f55c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f60d8009fb0 tx=0x7f60d8009b30).stop 2023-12-25T21:15:39.146 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.145+0000 7f60e759e700 1 -- 172.21.15.174:0/1714909986 shutdown_connections 2023-12-25T21:15:39.146 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.145+0000 7f60e759e700 1 --2- 172.21.15.174:0/1714909986 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60e80f51a0 0x7f60e80f55c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:39.146 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.145+0000 7f60e759e700 1 -- 172.21.15.174:0/1714909986 >> 172.21.15.174:0/1714909986 conn(0x7f60e80f09a0 msgr2=0x7f60e80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:39.147 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.145+0000 7f60e759e700 1 -- 172.21.15.174:0/1714909986 shutdown_connections 2023-12-25T21:15:39.147 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.145+0000 7f60e759e700 1 -- 172.21.15.174:0/1714909986 wait complete. 2023-12-25T21:15:39.147 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.145+0000 7f60e759e700 1 Processor -- start 2023-12-25T21:15:39.147 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.145+0000 7f60e759e700 1 -- start start 2023-12-25T21:15:39.147 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.145+0000 7f60e759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60e80f51a0 0x7f60e81053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:39.147 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.145+0000 7f60e759e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e81058a0 con 0x7f60e80f51a0 2023-12-25T21:15:39.148 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.146+0000 7f60e659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60e80f51a0 0x7f60e81053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:39.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.146+0000 7f60e659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60e80f51a0 0x7f60e81053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:56582/0 (socket says 172.21.15.174:56582) 2023-12-25T21:15:39.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.146+0000 7f60e659c700 1 -- 172.21.15.174:0/2138462664 learned_addr learned my addr 172.21.15.174:0/2138462664 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:15:39.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.146+0000 7f60e659c700 1 -- 172.21.15.174:0/2138462664 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d8005040 con 0x7f60e80f51a0 2023-12-25T21:15:39.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.146+0000 7f60e659c700 1 --2- 172.21.15.174:0/2138462664 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60e80f51a0 0x7f60e81053b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f60d8013510 tx=0x7f60d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:39.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.146+0000 7f60d6ffd700 1 -- 172.21.15.174:0/2138462664 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60d8009320 con 0x7f60e80f51a0 2023-12-25T21:15:39.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.147+0000 7f60d6ffd700 1 -- 172.21.15.174:0/2138462664 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f60d8013f00 con 0x7f60e80f51a0 2023-12-25T21:15:39.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.147+0000 7f60e759e700 1 -- 172.21.15.174:0/2138462664 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e8105aa0 con 0x7f60e80f51a0 2023-12-25T21:15:39.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.147+0000 7f60d6ffd700 1 -- 172.21.15.174:0/2138462664 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60d8014dc0 con 0x7f60e80f51a0 2023-12-25T21:15:39.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.147+0000 7f60e759e700 1 -- 172.21.15.174:0/2138462664 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e8105f30 con 0x7f60e80f51a0 2023-12-25T21:15:39.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.147+0000 7f60d6ffd700 1 -- 172.21.15.174:0/2138462664 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60d80095f0 con 0x7f60e80f51a0 2023-12-25T21:15:39.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.147+0000 7f60e759e700 1 -- 172.21.15.174:0/2138462664 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e8045ad0 con 0x7f60e80f51a0 2023-12-25T21:15:39.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.147+0000 7f60d6ffd700 1 --2- 172.21.15.174:0/2138462664 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f60d002ff60 0x7f60d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:39.149 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.147+0000 7f60d6ffd700 1 -- 172.21.15.174:0/2138462664 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f60d8041000 con 0x7f60e80f51a0 2023-12-25T21:15:39.150 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.148+0000 7f60e5d9b700 1 --2- 172.21.15.174:0/2138462664 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f60d002ff60 0x7f60d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:39.150 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.148+0000 7f60e5d9b700 1 --2- 172.21.15.174:0/2138462664 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f60d002ff60 0x7f60d0032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f60dc0096a0 tx=0x7f60dc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:39.153 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.153+0000 7f60d6ffd700 1 -- 172.21.15.174:0/2138462664 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60d801c790 con 0x7f60e80f51a0 2023-12-25T21:15:39.362 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.361+0000 7f60e759e700 1 -- 172.21.15.174:0/2138462664 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60e8106de0 con 0x7f60e80f51a0 2023-12-25T21:15:39.362 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.362+0000 7f60d6ffd700 1 -- 172.21.15.174:0/2138462664 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f60d8010230 con 0x7f60e80f51a0 2023-12-25T21:15:39.363 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:39.363 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:39.364 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.363+0000 7f60e759e700 1 -- 172.21.15.174:0/2138462664 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f60d002ff60 msgr2=0x7f60d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:39.365 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.364+0000 7f60e759e700 1 --2- 172.21.15.174:0/2138462664 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f60d002ff60 0x7f60d0032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f60dc0096a0 tx=0x7f60dc006b40).stop 2023-12-25T21:15:39.365 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.364+0000 7f60e759e700 1 -- 172.21.15.174:0/2138462664 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60e80f51a0 msgr2=0x7f60e81053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:39.365 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.364+0000 7f60e759e700 1 --2- 172.21.15.174:0/2138462664 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60e80f51a0 0x7f60e81053b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f60d8013510 tx=0x7f60d800b040).stop 2023-12-25T21:15:39.365 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.364+0000 7f60e759e700 1 -- 172.21.15.174:0/2138462664 shutdown_connections 2023-12-25T21:15:39.365 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.364+0000 7f60e759e700 1 --2- 172.21.15.174:0/2138462664 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f60d002ff60 0x7f60d0032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:39.365 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.364+0000 7f60e759e700 1 --2- 172.21.15.174:0/2138462664 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60e80f51a0 0x7f60e81053b0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:39.365 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.364+0000 7f60e759e700 1 -- 172.21.15.174:0/2138462664 >> 172.21.15.174:0/2138462664 conn(0x7f60e80f09a0 msgr2=0x7f60e80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:39.365 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.364+0000 7f60e759e700 1 -- 172.21.15.174:0/2138462664 shutdown_connections 2023-12-25T21:15:39.365 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:39.364+0000 7f60e759e700 1 -- 172.21.15.174:0/2138462664 wait complete. 2023-12-25T21:15:39.366 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:39.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:39 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:39.363293+0000 mon.smithi061 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.174:0/2138462664' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:15:40.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:40 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:38.651150+0000 mgr.smithi061.jxxeke (mgr.14164) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:40.736 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:15:40.737 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:15:41.889 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.884+0000 7fea99f97700 1 -- 172.21.15.174:0/1623207047 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea940f5330 msgr2=0x7fea940f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:41.893 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.884+0000 7fea99f97700 1 --2- 172.21.15.174:0/1623207047 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea940f5330 0x7fea940f5750 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fea88009fb0 tx=0x7fea88009b30).stop 2023-12-25T21:15:41.893 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.884+0000 7fea99f97700 1 -- 172.21.15.174:0/1623207047 shutdown_connections 2023-12-25T21:15:41.893 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.884+0000 7fea99f97700 1 --2- 172.21.15.174:0/1623207047 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea940f5330 0x7fea940f5750 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:41.893 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.884+0000 7fea99f97700 1 -- 172.21.15.174:0/1623207047 >> 172.21.15.174:0/1623207047 conn(0x7fea940f0a90 msgr2=0x7fea940f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:41.893 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.884+0000 7fea99f97700 1 -- 172.21.15.174:0/1623207047 shutdown_connections 2023-12-25T21:15:41.893 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.884+0000 7fea99f97700 1 -- 172.21.15.174:0/1623207047 wait complete. 2023-12-25T21:15:41.893 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.885+0000 7fea99f97700 1 Processor -- start 2023-12-25T21:15:41.894 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.885+0000 7fea99f97700 1 -- start start 2023-12-25T21:15:41.894 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.885+0000 7fea99f97700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea940f5330 0x7fea941096f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:41.894 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.885+0000 7fea99f97700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea94060220 con 0x7fea940f5330 2023-12-25T21:15:41.894 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.885+0000 7fea937fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea940f5330 0x7fea941096f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:41.894 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.885+0000 7fea937fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea940f5330 0x7fea941096f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:56598/0 (socket says 172.21.15.174:56598) 2023-12-25T21:15:41.894 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.885+0000 7fea937fe700 1 -- 172.21.15.174:0/438197294 learned_addr learned my addr 172.21.15.174:0/438197294 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:15:41.894 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.886+0000 7fea937fe700 1 -- 172.21.15.174:0/438197294 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea88005040 con 0x7fea940f5330 2023-12-25T21:15:41.894 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.886+0000 7fea937fe700 1 --2- 172.21.15.174:0/438197294 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea940f5330 0x7fea941096f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fea88013510 tx=0x7fea8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:41.894 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.886+0000 7fea7bfff700 1 -- 172.21.15.174:0/438197294 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea88009320 con 0x7fea940f5330 2023-12-25T21:15:41.894 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.886+0000 7fea99f97700 1 -- 172.21.15.174:0/438197294 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea94060420 con 0x7fea940f5330 2023-12-25T21:15:41.894 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.886+0000 7fea7bfff700 1 -- 172.21.15.174:0/438197294 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea88013f00 con 0x7fea940f5330 2023-12-25T21:15:41.894 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.886+0000 7fea7bfff700 1 -- 172.21.15.174:0/438197294 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea88014d90 con 0x7fea940f5330 2023-12-25T21:15:41.895 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.886+0000 7fea99f97700 1 -- 172.21.15.174:0/438197294 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea94109da0 con 0x7fea940f5330 2023-12-25T21:15:41.895 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.887+0000 7fea7bfff700 1 -- 172.21.15.174:0/438197294 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea880263a0 con 0x7fea940f5330 2023-12-25T21:15:41.895 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.887+0000 7fea7bfff700 1 --2- 172.21.15.174:0/438197294 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fea7c02ff60 0x7fea7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:41.895 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.887+0000 7fea7bfff700 1 -- 172.21.15.174:0/438197294 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea880410c0 con 0x7fea940f5330 2023-12-25T21:15:41.895 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.887+0000 7fea99f97700 1 -- 172.21.15.174:0/438197294 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea94045ad0 con 0x7fea940f5330 2023-12-25T21:15:41.895 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.887+0000 7fea92ffd700 1 --2- 172.21.15.174:0/438197294 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fea7c02ff60 0x7fea7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:41.895 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.888+0000 7fea92ffd700 1 --2- 172.21.15.174:0/438197294 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fea7c02ff60 0x7fea7c032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fea840096a0 tx=0x7fea84006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:41.896 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:41.892+0000 7fea7bfff700 1 -- 172.21.15.174:0/438197294 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea88012920 con 0x7fea940f5330 2023-12-25T21:15:42.098 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:42.097+0000 7fea99f97700 1 -- 172.21.15.174:0/438197294 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea940faf30 con 0x7fea940f5330 2023-12-25T21:15:42.099 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:42.098+0000 7fea7bfff700 1 -- 172.21.15.174:0/438197294 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fea8802b020 con 0x7fea940f5330 2023-12-25T21:15:42.099 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:42.099 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:42.101 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:42.100+0000 7fea99f97700 1 -- 172.21.15.174:0/438197294 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fea7c02ff60 msgr2=0x7fea7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:42.101 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:42.100+0000 7fea99f97700 1 --2- 172.21.15.174:0/438197294 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fea7c02ff60 0x7fea7c032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fea840096a0 tx=0x7fea84006b40).stop 2023-12-25T21:15:42.101 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:42.100+0000 7fea99f97700 1 -- 172.21.15.174:0/438197294 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea940f5330 msgr2=0x7fea941096f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:42.101 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:42.100+0000 7fea99f97700 1 --2- 172.21.15.174:0/438197294 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea940f5330 0x7fea941096f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fea88013510 tx=0x7fea8800b040).stop 2023-12-25T21:15:42.101 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:42.100+0000 7fea99f97700 1 -- 172.21.15.174:0/438197294 shutdown_connections 2023-12-25T21:15:42.102 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:42.100+0000 7fea99f97700 1 --2- 172.21.15.174:0/438197294 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fea7c02ff60 0x7fea7c032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:42.102 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:42.100+0000 7fea99f97700 1 --2- 172.21.15.174:0/438197294 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea940f5330 0x7fea941096f0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:42.102 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:42.100+0000 7fea99f97700 1 -- 172.21.15.174:0/438197294 >> 172.21.15.174:0/438197294 conn(0x7fea940f0a90 msgr2=0x7fea940f1530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:42.102 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:42.100+0000 7fea99f97700 1 -- 172.21.15.174:0/438197294 shutdown_connections 2023-12-25T21:15:42.102 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:42.100+0000 7fea99f97700 1 -- 172.21.15.174:0/438197294 wait complete. 2023-12-25T21:15:42.103 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:42.709 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:42 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:40.651552+0000 mgr.smithi061.jxxeke (mgr.14164) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:42.710 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:42 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:42.099617+0000 mon.smithi061 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.174:0/438197294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:15:43.481 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:15:43.481 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:15:44.649 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.648+0000 7fcf2b859700 1 -- 172.21.15.174:0/1212573757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf240f6510 msgr2=0x7fcf240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:44.649 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.648+0000 7fcf2b859700 1 --2- 172.21.15.174:0/1212573757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf240f6510 0x7fcf240f6930 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fcf14009fb0 tx=0x7fcf14009b30).stop 2023-12-25T21:15:44.649 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.648+0000 7fcf2b859700 1 -- 172.21.15.174:0/1212573757 shutdown_connections 2023-12-25T21:15:44.649 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.648+0000 7fcf2b859700 1 --2- 172.21.15.174:0/1212573757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf240f6510 0x7fcf240f6930 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:44.649 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.648+0000 7fcf2b859700 1 -- 172.21.15.174:0/1212573757 >> 172.21.15.174:0/1212573757 conn(0x7fcf240f1c70 msgr2=0x7fcf240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:44.649 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.648+0000 7fcf2b859700 1 -- 172.21.15.174:0/1212573757 shutdown_connections 2023-12-25T21:15:44.649 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.648+0000 7fcf2b859700 1 -- 172.21.15.174:0/1212573757 wait complete. 2023-12-25T21:15:44.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.648+0000 7fcf2b859700 1 Processor -- start 2023-12-25T21:15:44.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.649+0000 7fcf2b859700 1 -- start start 2023-12-25T21:15:44.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.649+0000 7fcf2b859700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf240f6510 0x7fcf2406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:44.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.649+0000 7fcf2b859700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf24069e20 con 0x7fcf240f6510 2023-12-25T21:15:44.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.649+0000 7fcf295f5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf240f6510 0x7fcf2406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:44.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.649+0000 7fcf295f5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf240f6510 0x7fcf2406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:56620/0 (socket says 172.21.15.174:56620) 2023-12-25T21:15:44.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.649+0000 7fcf295f5700 1 -- 172.21.15.174:0/3934225012 learned_addr learned my addr 172.21.15.174:0/3934225012 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:15:44.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.649+0000 7fcf295f5700 1 -- 172.21.15.174:0/3934225012 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf14005040 con 0x7fcf240f6510 2023-12-25T21:15:44.651 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.649+0000 7fcf295f5700 1 --2- 172.21.15.174:0/3934225012 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf240f6510 0x7fcf2406b9a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fcf14014000 tx=0x7fcf1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:44.652 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.650+0000 7fcf19ffb700 1 -- 172.21.15.174:0/3934225012 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf140095f0 con 0x7fcf240f6510 2023-12-25T21:15:44.652 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.650+0000 7fcf2b859700 1 -- 172.21.15.174:0/3934225012 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf24069fc0 con 0x7fcf240f6510 2023-12-25T21:15:44.652 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.650+0000 7fcf19ffb700 1 -- 172.21.15.174:0/3934225012 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf14007e70 con 0x7fcf240f6510 2023-12-25T21:15:44.652 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.650+0000 7fcf19ffb700 1 -- 172.21.15.174:0/3934225012 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf14014ea0 con 0x7fcf240f6510 2023-12-25T21:15:44.652 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.650+0000 7fcf2b859700 1 -- 172.21.15.174:0/3934225012 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2406a450 con 0x7fcf240f6510 2023-12-25T21:15:44.653 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.651+0000 7fcf19ffb700 1 -- 172.21.15.174:0/3934225012 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcf14025070 con 0x7fcf240f6510 2023-12-25T21:15:44.653 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.651+0000 7fcf2b859700 1 -- 172.21.15.174:0/3934225012 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf24059100 con 0x7fcf240f6510 2023-12-25T21:15:44.653 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.651+0000 7fcf19ffb700 1 --2- 172.21.15.174:0/3934225012 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fcf1002ff60 0x7fcf10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:44.653 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.651+0000 7fcf19ffb700 1 -- 172.21.15.174:0/3934225012 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcf14040a00 con 0x7fcf240f6510 2023-12-25T21:15:44.653 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.651+0000 7fcf28df4700 1 --2- 172.21.15.174:0/3934225012 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fcf1002ff60 0x7fcf10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:44.653 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.651+0000 7fcf28df4700 1 --2- 172.21.15.174:0/3934225012 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fcf1002ff60 0x7fcf10032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fcf200096a0 tx=0x7fcf20006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:44.656 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.656+0000 7fcf19ffb700 1 -- 172.21.15.174:0/3934225012 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf140079c0 con 0x7fcf240f6510 2023-12-25T21:15:44.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:44 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:42.651897+0000 mgr.smithi061.jxxeke (mgr.14164) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:44.864 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.863+0000 7fcf2b859700 1 -- 172.21.15.174:0/3934225012 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcf240fbf60 con 0x7fcf240f6510 2023-12-25T21:15:44.864 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.863+0000 7fcf19ffb700 1 -- 172.21.15.174:0/3934225012 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcf1400f300 con 0x7fcf240f6510 2023-12-25T21:15:44.865 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:44.865 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:44.866 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.865+0000 7fcf2b859700 1 -- 172.21.15.174:0/3934225012 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fcf1002ff60 msgr2=0x7fcf10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:44.866 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.865+0000 7fcf2b859700 1 --2- 172.21.15.174:0/3934225012 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fcf1002ff60 0x7fcf10032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fcf200096a0 tx=0x7fcf20006b40).stop 2023-12-25T21:15:44.866 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.865+0000 7fcf2b859700 1 -- 172.21.15.174:0/3934225012 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf240f6510 msgr2=0x7fcf2406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:44.866 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.865+0000 7fcf2b859700 1 --2- 172.21.15.174:0/3934225012 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf240f6510 0x7fcf2406b9a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fcf14014000 tx=0x7fcf1400b040).stop 2023-12-25T21:15:44.866 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.865+0000 7fcf2b859700 1 -- 172.21.15.174:0/3934225012 shutdown_connections 2023-12-25T21:15:44.866 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.865+0000 7fcf2b859700 1 --2- 172.21.15.174:0/3934225012 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fcf1002ff60 0x7fcf10032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:44.866 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.865+0000 7fcf2b859700 1 --2- 172.21.15.174:0/3934225012 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf240f6510 0x7fcf2406b9a0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:44.867 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.865+0000 7fcf2b859700 1 -- 172.21.15.174:0/3934225012 >> 172.21.15.174:0/3934225012 conn(0x7fcf240f1c70 msgr2=0x7fcf240f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:44.867 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.865+0000 7fcf2b859700 1 -- 172.21.15.174:0/3934225012 shutdown_connections 2023-12-25T21:15:44.867 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:44.865+0000 7fcf2b859700 1 -- 172.21.15.174:0/3934225012 wait complete. 2023-12-25T21:15:44.867 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:45.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:45 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:44.865085+0000 mon.smithi061 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.174:0/3934225012' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:15:46.261 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:15:46.262 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:15:46.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:46 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:44.652124+0000 mgr.smithi061.jxxeke (mgr.14164) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:47.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.577+0000 7f7d8c9b6700 1 -- 172.21.15.174:0/1988364319 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d880f5410 msgr2=0x7f7d880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:47.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.577+0000 7f7d8c9b6700 1 --2- 172.21.15.174:0/1988364319 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d880f5410 0x7f7d880f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f7d7c009fb0 tx=0x7f7d7c009b30).stop 2023-12-25T21:15:47.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.577+0000 7f7d8c9b6700 1 -- 172.21.15.174:0/1988364319 shutdown_connections 2023-12-25T21:15:47.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.577+0000 7f7d8c9b6700 1 --2- 172.21.15.174:0/1988364319 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d880f5410 0x7f7d880f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:47.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.577+0000 7f7d8c9b6700 1 -- 172.21.15.174:0/1988364319 >> 172.21.15.174:0/1988364319 conn(0x7f7d880f0b70 msgr2=0x7f7d880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:47.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.577+0000 7f7d8c9b6700 1 -- 172.21.15.174:0/1988364319 shutdown_connections 2023-12-25T21:15:47.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.577+0000 7f7d8c9b6700 1 -- 172.21.15.174:0/1988364319 wait complete. 2023-12-25T21:15:47.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.578+0000 7f7d8c9b6700 1 Processor -- start 2023-12-25T21:15:47.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.578+0000 7f7d8c9b6700 1 -- start start 2023-12-25T21:15:47.579 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.578+0000 7f7d8c9b6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d880f5410 0x7f7d881066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:47.580 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.578+0000 7f7d8c9b6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d88106bd0 con 0x7f7d880f5410 2023-12-25T21:15:47.580 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.578+0000 7f7d8659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d880f5410 0x7f7d881066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:47.580 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.578+0000 7f7d8659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d880f5410 0x7f7d881066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:33222/0 (socket says 172.21.15.174:33222) 2023-12-25T21:15:47.580 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.578+0000 7f7d8659c700 1 -- 172.21.15.174:0/2160273538 learned_addr learned my addr 172.21.15.174:0/2160273538 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:15:47.581 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.578+0000 7f7d8659c700 1 -- 172.21.15.174:0/2160273538 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7c005040 con 0x7f7d880f5410 2023-12-25T21:15:47.581 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.579+0000 7f7d8659c700 1 --2- 172.21.15.174:0/2160273538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d880f5410 0x7f7d881066e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f7d7c014000 tx=0x7f7d7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:47.581 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.579+0000 7f7d76ffd700 1 -- 172.21.15.174:0/2160273538 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d7c0095f0 con 0x7f7d880f5410 2023-12-25T21:15:47.581 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.579+0000 7f7d8c9b6700 1 -- 172.21.15.174:0/2160273538 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d88106dd0 con 0x7f7d880f5410 2023-12-25T21:15:47.581 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.579+0000 7f7d76ffd700 1 -- 172.21.15.174:0/2160273538 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d7c007e70 con 0x7f7d880f5410 2023-12-25T21:15:47.582 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.579+0000 7f7d76ffd700 1 -- 172.21.15.174:0/2160273538 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d7c014ea0 con 0x7f7d880f5410 2023-12-25T21:15:47.582 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.579+0000 7f7d8c9b6700 1 -- 172.21.15.174:0/2160273538 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d88107260 con 0x7f7d880f5410 2023-12-25T21:15:47.582 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.580+0000 7f7d76ffd700 1 -- 172.21.15.174:0/2160273538 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7d7c025070 con 0x7f7d880f5410 2023-12-25T21:15:47.582 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.580+0000 7f7d8c9b6700 1 -- 172.21.15.174:0/2160273538 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d88045ad0 con 0x7f7d880f5410 2023-12-25T21:15:47.582 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.580+0000 7f7d76ffd700 1 --2- 172.21.15.174:0/2160273538 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7d7002ff60 0x7f7d70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:47.582 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.580+0000 7f7d76ffd700 1 -- 172.21.15.174:0/2160273538 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7d7c040980 con 0x7f7d880f5410 2023-12-25T21:15:47.582 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.580+0000 7f7d85d9b700 1 --2- 172.21.15.174:0/2160273538 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7d7002ff60 0x7f7d70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:47.582 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.580+0000 7f7d85d9b700 1 --2- 172.21.15.174:0/2160273538 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7d7002ff60 0x7f7d70032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f7d780096a0 tx=0x7f7d78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:47.585 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.585+0000 7f7d76ffd700 1 -- 172.21.15.174:0/2160273538 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d7c0079c0 con 0x7f7d880f5410 2023-12-25T21:15:47.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:47 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:46.652488+0000 mgr.smithi061.jxxeke (mgr.14164) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:47.792 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.791+0000 7f7d8c9b6700 1 -- 172.21.15.174:0/2160273538 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d880fae60 con 0x7f7d880f5410 2023-12-25T21:15:47.792 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.792+0000 7f7d76ffd700 1 -- 172.21.15.174:0/2160273538 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7d7c00f300 con 0x7f7d880f5410 2023-12-25T21:15:47.793 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:47.793 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:47.794 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.793+0000 7f7d8c9b6700 1 -- 172.21.15.174:0/2160273538 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7d7002ff60 msgr2=0x7f7d70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:47.794 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.793+0000 7f7d8c9b6700 1 --2- 172.21.15.174:0/2160273538 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7d7002ff60 0x7f7d70032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f7d780096a0 tx=0x7f7d78006b40).stop 2023-12-25T21:15:47.794 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.793+0000 7f7d8c9b6700 1 -- 172.21.15.174:0/2160273538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d880f5410 msgr2=0x7f7d881066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:47.795 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.793+0000 7f7d8c9b6700 1 --2- 172.21.15.174:0/2160273538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d880f5410 0x7f7d881066e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f7d7c014000 tx=0x7f7d7c00b040).stop 2023-12-25T21:15:47.795 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.793+0000 7f7d8c9b6700 1 -- 172.21.15.174:0/2160273538 shutdown_connections 2023-12-25T21:15:47.795 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.793+0000 7f7d8c9b6700 1 --2- 172.21.15.174:0/2160273538 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f7d7002ff60 0x7f7d70032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:47.795 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.793+0000 7f7d8c9b6700 1 --2- 172.21.15.174:0/2160273538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d880f5410 0x7f7d881066e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:47.795 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.793+0000 7f7d8c9b6700 1 -- 172.21.15.174:0/2160273538 >> 172.21.15.174:0/2160273538 conn(0x7f7d880f0b70 msgr2=0x7f7d8805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:47.795 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.793+0000 7f7d8c9b6700 1 -- 172.21.15.174:0/2160273538 shutdown_connections 2023-12-25T21:15:47.795 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:47.793+0000 7f7d8c9b6700 1 -- 172.21.15.174:0/2160273538 wait complete. 2023-12-25T21:15:47.795 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:48.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:48 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:47.793351+0000 mon.smithi061 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.174:0/2160273538' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:15:49.173 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:15:49.173 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:15:49.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:49 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:48.652807+0000 mgr.smithi061.jxxeke (mgr.14164) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:50.460 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.457+0000 7fddd3300700 1 -- 172.21.15.174:0/616785660 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddcc0f62a0 msgr2=0x7fddcc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:50.460 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.457+0000 7fddd3300700 1 --2- 172.21.15.174:0/616785660 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddcc0f62a0 0x7fddcc0f66c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fddc8009fb0 tx=0x7fddc8009b30).stop 2023-12-25T21:15:50.460 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.457+0000 7fddd3300700 1 -- 172.21.15.174:0/616785660 shutdown_connections 2023-12-25T21:15:50.460 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.457+0000 7fddd3300700 1 --2- 172.21.15.174:0/616785660 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddcc0f62a0 0x7fddcc0f66c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:50.460 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.457+0000 7fddd3300700 1 -- 172.21.15.174:0/616785660 >> 172.21.15.174:0/616785660 conn(0x7fddcc0f1aa0 msgr2=0x7fddcc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:50.460 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.457+0000 7fddd3300700 1 -- 172.21.15.174:0/616785660 shutdown_connections 2023-12-25T21:15:50.460 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.457+0000 7fddd3300700 1 -- 172.21.15.174:0/616785660 wait complete. 2023-12-25T21:15:50.460 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.458+0000 7fddd3300700 1 Processor -- start 2023-12-25T21:15:50.460 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.458+0000 7fddd3300700 1 -- start start 2023-12-25T21:15:50.460 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.458+0000 7fddd3300700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddcc0f62a0 0x7fddcc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:50.461 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.458+0000 7fddd3300700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc069e20 con 0x7fddcc0f62a0 2023-12-25T21:15:50.461 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.458+0000 7fddd22fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddcc0f62a0 0x7fddcc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:50.461 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.458+0000 7fddd22fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddcc0f62a0 0x7fddcc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:33236/0 (socket says 172.21.15.174:33236) 2023-12-25T21:15:50.461 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.458+0000 7fddd22fe700 1 -- 172.21.15.174:0/3851561312 learned_addr learned my addr 172.21.15.174:0/3851561312 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:15:50.461 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.458+0000 7fddd22fe700 1 -- 172.21.15.174:0/3851561312 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc8005040 con 0x7fddcc0f62a0 2023-12-25T21:15:50.461 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.459+0000 7fddd22fe700 1 --2- 172.21.15.174:0/3851561312 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddcc0f62a0 0x7fddcc06b9a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fddc8009830 tx=0x7fddc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:50.461 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.459+0000 7fddc2ffd700 1 -- 172.21.15.174:0/3851561312 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddc8009320 con 0x7fddcc0f62a0 2023-12-25T21:15:50.461 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.459+0000 7fddd3300700 1 -- 172.21.15.174:0/3851561312 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddcc069fc0 con 0x7fddcc0f62a0 2023-12-25T21:15:50.461 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.459+0000 7fddc2ffd700 1 -- 172.21.15.174:0/3851561312 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fddc80136c0 con 0x7fddcc0f62a0 2023-12-25T21:15:50.461 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.459+0000 7fddc2ffd700 1 -- 172.21.15.174:0/3851561312 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddc801ee90 con 0x7fddcc0f62a0 2023-12-25T21:15:50.461 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.459+0000 7fddd3300700 1 -- 172.21.15.174:0/3851561312 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddcc06a450 con 0x7fddcc0f62a0 2023-12-25T21:15:50.462 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.460+0000 7fddc2ffd700 1 -- 172.21.15.174:0/3851561312 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fddc801e420 con 0x7fddcc0f62a0 2023-12-25T21:15:50.462 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.460+0000 7fddd3300700 1 -- 172.21.15.174:0/3851561312 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddcc059100 con 0x7fddcc0f62a0 2023-12-25T21:15:50.462 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.460+0000 7fddc2ffd700 1 --2- 172.21.15.174:0/3851561312 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fddb802ff10 0x7fddb80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:50.462 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.460+0000 7fddc2ffd700 1 -- 172.21.15.174:0/3851561312 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fddc8041010 con 0x7fddcc0f62a0 2023-12-25T21:15:50.463 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.460+0000 7fddd1afd700 1 --2- 172.21.15.174:0/3851561312 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fddb802ff10 0x7fddb80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:50.463 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.460+0000 7fddd1afd700 1 --2- 172.21.15.174:0/3851561312 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fddb802ff10 0x7fddb80323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fddbc0096a0 tx=0x7fddbc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:50.465 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.465+0000 7fddc2ffd700 1 -- 172.21.15.174:0/3851561312 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fddc80095f0 con 0x7fddcc0f62a0 2023-12-25T21:15:50.671 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.670+0000 7fddd3300700 1 -- 172.21.15.174:0/3851561312 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fddcc0fbce0 con 0x7fddcc0f62a0 2023-12-25T21:15:50.671 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.671+0000 7fddc2ffd700 1 -- 172.21.15.174:0/3851561312 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fddc801cb10 con 0x7fddcc0f62a0 2023-12-25T21:15:50.672 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:50.672 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:50.673 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.672+0000 7fddd3300700 1 -- 172.21.15.174:0/3851561312 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fddb802ff10 msgr2=0x7fddb80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:50.673 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.672+0000 7fddd3300700 1 --2- 172.21.15.174:0/3851561312 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fddb802ff10 0x7fddb80323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fddbc0096a0 tx=0x7fddbc006b40).stop 2023-12-25T21:15:50.673 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.672+0000 7fddd3300700 1 -- 172.21.15.174:0/3851561312 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddcc0f62a0 msgr2=0x7fddcc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:50.674 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.672+0000 7fddd3300700 1 --2- 172.21.15.174:0/3851561312 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddcc0f62a0 0x7fddcc06b9a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fddc8009830 tx=0x7fddc800b040).stop 2023-12-25T21:15:50.674 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.672+0000 7fddd3300700 1 -- 172.21.15.174:0/3851561312 shutdown_connections 2023-12-25T21:15:50.674 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.672+0000 7fddd3300700 1 --2- 172.21.15.174:0/3851561312 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7fddb802ff10 0x7fddb80323d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:50.674 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.672+0000 7fddd3300700 1 --2- 172.21.15.174:0/3851561312 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddcc0f62a0 0x7fddcc06b9a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:50.674 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.672+0000 7fddd3300700 1 -- 172.21.15.174:0/3851561312 >> 172.21.15.174:0/3851561312 conn(0x7fddcc0f1aa0 msgr2=0x7fddcc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:50.674 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.672+0000 7fddd3300700 1 -- 172.21.15.174:0/3851561312 shutdown_connections 2023-12-25T21:15:50.674 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:50.672+0000 7fddd3300700 1 -- 172.21.15.174:0/3851561312 wait complete. 2023-12-25T21:15:50.674 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:51.205 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:50 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:50.672200+0000 mon.smithi061 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.174:0/3851561312' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-25T21:15:52.097 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-25T21:15:52.097 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5532d712-a369-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-25T21:15:52.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:52 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: cluster 2023-12-25T21:15:50.653095+0000 mgr.smithi061.jxxeke (mgr.14164) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-25T21:15:53.283 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.280+0000 7f4006f3f700 1 -- 172.21.15.174:0/2592111527 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40000f51a0 msgr2=0x7f40000f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:53.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.280+0000 7f4006f3f700 1 --2- 172.21.15.174:0/2592111527 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40000f51a0 0x7f40000f55c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f3ffc009fb0 tx=0x7f3ffc009b30).stop 2023-12-25T21:15:53.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.280+0000 7f4006f3f700 1 -- 172.21.15.174:0/2592111527 shutdown_connections 2023-12-25T21:15:53.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.280+0000 7f4006f3f700 1 --2- 172.21.15.174:0/2592111527 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40000f51a0 0x7f40000f55c0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:53.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.280+0000 7f4006f3f700 1 -- 172.21.15.174:0/2592111527 >> 172.21.15.174:0/2592111527 conn(0x7f40000f09a0 msgr2=0x7f40000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:53.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.280+0000 7f4006f3f700 1 -- 172.21.15.174:0/2592111527 shutdown_connections 2023-12-25T21:15:53.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.280+0000 7f4006f3f700 1 -- 172.21.15.174:0/2592111527 wait complete. 2023-12-25T21:15:53.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.280+0000 7f4006f3f700 1 Processor -- start 2023-12-25T21:15:53.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.280+0000 7f4006f3f700 1 -- start start 2023-12-25T21:15:53.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.280+0000 7f4006f3f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40000f51a0 0x7f4000109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:53.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.280+0000 7f4006f3f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4000060280 con 0x7f40000f51a0 2023-12-25T21:15:53.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.281+0000 7f4005f3d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40000f51a0 0x7f4000109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:53.284 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.281+0000 7f4005f3d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40000f51a0 0x7f4000109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.174:33256/0 (socket says 172.21.15.174:33256) 2023-12-25T21:15:53.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.281+0000 7f4005f3d700 1 -- 172.21.15.174:0/552931106 learned_addr learned my addr 172.21.15.174:0/552931106 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-12-25T21:15:53.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.281+0000 7f4005f3d700 1 -- 172.21.15.174:0/552931106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ffc005040 con 0x7f40000f51a0 2023-12-25T21:15:53.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.281+0000 7f4005f3d700 1 --2- 172.21.15.174:0/552931106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40000f51a0 0x7f4000109170 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f3ffc013510 tx=0x7f3ffc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:53.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.281+0000 7f3ff67fc700 1 -- 172.21.15.174:0/552931106 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ffc009320 con 0x7f40000f51a0 2023-12-25T21:15:53.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.281+0000 7f4006f3f700 1 -- 172.21.15.174:0/552931106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4000060480 con 0x7f40000f51a0 2023-12-25T21:15:53.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.281+0000 7f3ff67fc700 1 -- 172.21.15.174:0/552931106 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ffc013f00 con 0x7f40000f51a0 2023-12-25T21:15:53.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.281+0000 7f3ff67fc700 1 -- 172.21.15.174:0/552931106 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ffc014dc0 con 0x7f40000f51a0 2023-12-25T21:15:53.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.281+0000 7f4006f3f700 1 -- 172.21.15.174:0/552931106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40001098a0 con 0x7f40000f51a0 2023-12-25T21:15:53.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.282+0000 7f3ff67fc700 1 -- 172.21.15.174:0/552931106 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ffc0095f0 con 0x7f40000f51a0 2023-12-25T21:15:53.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.282+0000 7f4006f3f700 1 -- 172.21.15.174:0/552931106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4000045ad0 con 0x7f40000f51a0 2023-12-25T21:15:53.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.282+0000 7f3ff67fc700 1 --2- 172.21.15.174:0/552931106 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3fec02ff60 0x7f3fec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-25T21:15:53.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.282+0000 7f3ff67fc700 1 -- 172.21.15.174:0/552931106 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ffc0410c0 con 0x7f40000f51a0 2023-12-25T21:15:53.285 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.282+0000 7f400573c700 1 --2- 172.21.15.174:0/552931106 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3fec02ff60 0x7f3fec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-25T21:15:53.286 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.283+0000 7f400573c700 1 --2- 172.21.15.174:0/552931106 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3fec02ff60 0x7f3fec032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f3ff00096a0 tx=0x7f3ff0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-25T21:15:53.288 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.287+0000 7f3ff67fc700 1 -- 172.21.15.174:0/552931106 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ffc014420 con 0x7f40000f51a0 2023-12-25T21:15:53.492 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.491+0000 7f4006f3f700 1 -- 172.21.15.174:0/552931106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40000fabe0 con 0x7f40000f51a0 2023-12-25T21:15:53.492 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.492+0000 7f3ff67fc700 1 -- 172.21.15.174:0/552931106 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3ffc010230 con 0x7f40000f51a0 2023-12-25T21:15:53.493 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-12-25T21:15:53.493 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"5532d712-a369-11ee-95a5-87774f69a715","modified":"2023-12-25T21:05:56.011996Z","created":"2023-12-25T21:05:56.011996Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi061","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:3300","nonce":0},{"type":"v1","addr":"172.21.15.61:6789","nonce":0}]},"addr":"172.21.15.61:6789/0","public_addr":"172.21.15.61:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-25T21:15:53.494 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.493+0000 7f4006f3f700 1 -- 172.21.15.174:0/552931106 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3fec02ff60 msgr2=0x7f3fec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:53.494 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.493+0000 7f4006f3f700 1 --2- 172.21.15.174:0/552931106 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3fec02ff60 0x7f3fec032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f3ff00096a0 tx=0x7f3ff0006b40).stop 2023-12-25T21:15:53.494 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.493+0000 7f4006f3f700 1 -- 172.21.15.174:0/552931106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40000f51a0 msgr2=0x7f4000109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-25T21:15:53.494 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.493+0000 7f4006f3f700 1 --2- 172.21.15.174:0/552931106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40000f51a0 0x7f4000109170 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f3ffc013510 tx=0x7f3ffc00b040).stop 2023-12-25T21:15:53.494 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.493+0000 7f4006f3f700 1 -- 172.21.15.174:0/552931106 shutdown_connections 2023-12-25T21:15:53.495 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.493+0000 7f4006f3f700 1 --2- 172.21.15.174:0/552931106 >> [v2:172.21.15.61:6800/889227411,v1:172.21.15.61:6801/889227411] conn(0x7f3fec02ff60 0x7f3fec032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:53.495 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.493+0000 7f4006f3f700 1 --2- 172.21.15.174:0/552931106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40000f51a0 0x7f4000109170 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-25T21:15:53.495 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.493+0000 7f4006f3f700 1 -- 172.21.15.174:0/552931106 >> 172.21.15.174:0/552931106 conn(0x7f40000f09a0 msgr2=0x7f40000f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-25T21:15:53.495 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.493+0000 7f4006f3f700 1 -- 172.21.15.174:0/552931106 shutdown_connections 2023-12-25T21:15:53.495 INFO:teuthology.orchestra.run.smithi174.stderr:2023-12-25T21:15:53.493+0000 7f4006f3f700 1 -- 172.21.15.174:0/552931106 wait complete. 2023-12-25T21:15:53.495 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2023-12-25T21:15:53.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:52.710781+0000 mon.smithi061 (mon.0) 326 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/mirror_snapshot_schedule"}]: dispatch 2023-12-25T21:15:53.705 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:53 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: audit 2023-12-25T21:15:52.711066+0000 mon.smithi061 (mon.0) 327 : audit [INF] from='mgr.14164 172.21.15.61:0/4075510542' entity='mgr.smithi061.jxxeke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi061.jxxeke/trash_purge_schedule"}]: dispatch 2023-12-25T21:15:53.875 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2023-12-25T21:15:53.877 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-12-25T21:15:53.877 DEBUG:teuthology.orchestra.run.smithi061:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-25T21:15:53.902 DEBUG:teuthology.orchestra.run.smithi174:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-25T21:15:53.920 INFO:tasks.cephadm:Stopping all daemons... 2023-12-25T21:15:53.920 INFO:tasks.cephadm.mon.smithi061:Stopping mon.smithi061... 2023-12-25T21:15:53.920 DEBUG:teuthology.orchestra.run.smithi061:> sudo systemctl stop ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi061 2023-12-25T21:15:54.289 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:53 smithi061 systemd[1]: Stopping Ceph mon.smithi061 for 5532d712-a369-11ee-95a5-87774f69a715... 2023-12-25T21:15:54.289 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:54 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:15:54.054+0000 7f833c382700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi061 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-25T21:15:54.289 INFO:journalctl@ceph.mon.smithi061.smithi061.stdout:Dec 25 21:15:54 smithi061 ceph-5532d712-a369-11ee-95a5-87774f69a715-mon.smithi061[111722]: debug 2023-12-25T21:15:54.054+0000 7f833c382700 -1 mon.smithi061@0(leader) e1 *** Got Signal Terminated *** 2023-12-25T21:15:54.493 DEBUG:teuthology.orchestra.run.smithi061:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi061.service' 2023-12-25T21:15:54.540 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-25T21:15:54.540 INFO:tasks.cephadm.mon.smithi061:Stopped mon.smithi061 2023-12-25T21:15:54.541 INFO:tasks.cephadm.mon.smithi174:Stopping mon.smithi174... 2023-12-25T21:15:54.541 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi174 2023-12-25T21:15:54.574 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi174.service: Unit ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi174.service not loaded. 2023-12-25T21:15:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-12-25T21:15:54.576 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi174" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi174 with status 5: 'sudo systemctl stop ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi174' 2023-12-25T21:15:54.577 INFO:tasks.cephadm:Archiving crash dumps... 2023-12-25T21:15:54.579 DEBUG:teuthology.misc:Transferring archived files from smithi061:/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500977/remote/ubuntu@smithi061.front.sepia.ceph.com/crash 2023-12-25T21:15:54.579 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar c -f - -C /var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/crash -- . 2023-12-25T21:15:54.615 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500977/remote/ubuntu@smithi174.front.sepia.ceph.com/crash 2023-12-25T21:15:54.615 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/crash -- . 2023-12-25T21:15:54.682 INFO:teuthology.orchestra.run.smithi174.stderr:tar: /var/lib/ceph/5532d712-a369-11ee-95a5-87774f69a715/crash: Cannot open: No such file or directory 2023-12-25T21:15:54.682 INFO:teuthology.orchestra.run.smithi174.stderr:tar: Error is not recoverable: exiting now 2023-12-25T21:15:54.683 INFO:tasks.cephadm:Checking cluster log for badness... 2023-12-25T21:15:54.683 DEBUG:teuthology.orchestra.run.smithi061:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-25T21:15:54.715 INFO:teuthology.orchestra.run.smithi061.stderr:grep: /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715/ceph.log: No such file or directory 2023-12-25T21:15:54.716 INFO:tasks.cephadm:Compressing logs... 2023-12-25T21:15:54.717 DEBUG:teuthology.orchestra.run.smithi061:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-25T21:15:54.760 DEBUG:teuthology.orchestra.run.smithi174:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-25T21:15:54.786 INFO:teuthology.orchestra.run.smithi061.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-12-25T21:15:54.786 INFO:teuthology.orchestra.run.smithi061.stderr:: No such file or directory 2023-12-25T21:15:54.786 INFO:teuthology.orchestra.run.smithi061.stderr:gzip -5 --verbose -- /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715/ceph-volume.log 2023-12-25T21:15:54.787 INFO:teuthology.orchestra.run.smithi174.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-12-25T21:15:54.787 INFO:teuthology.orchestra.run.smithi174.stderr:: No such file or directory 2023-12-25T21:15:54.787 INFO:teuthology.orchestra.run.smithi061.stderr:gzip -5 --verbose -- /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715/ceph-mon.smithi061.log 2023-12-25T21:15:54.788 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715/ceph-mgr.smithi061.jxxeke.log 2023-12-25T21:15:54.788 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715/ceph-volume.log: /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715/ceph-mon.smithi061.log: /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715/ceph-mgr.smithi061.jxxeke.log: 88.3% -- replaced with /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715/ceph-volume.log.gz 2023-12-25T21:15:54.789 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/cephadm.log: 96.1% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-25T21:15:54.790 INFO:teuthology.orchestra.run.smithi174.stderr: 2023-12-25T21:15:54.790 INFO:teuthology.orchestra.run.smithi174.stderr:real 0m0.013s 2023-12-25T21:15:54.790 INFO:teuthology.orchestra.run.smithi174.stderr:user 0m0.009s 2023-12-25T21:15:54.792 INFO:teuthology.orchestra.run.smithi174.stderr:sys 0m0.012s 2023-12-25T21:15:54.792 INFO:teuthology.orchestra.run.smithi061.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-25T21:15:54.866 INFO:teuthology.orchestra.run.smithi061.stderr:gzip: /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715/ceph-mgr.smithi061.jxxeke.log: file size changed while zipping 2023-12-25T21:15:54.866 INFO:teuthology.orchestra.run.smithi061.stderr: 92.6% -- replaced with /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715/ceph-mgr.smithi061.jxxeke.log.gz 2023-12-25T21:15:54.936 INFO:teuthology.orchestra.run.smithi061.stderr: 90.6% -- replaced with /var/log/ceph/5532d712-a369-11ee-95a5-87774f69a715/ceph-mon.smithi061.log.gz 2023-12-25T21:15:54.938 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-12-25T21:15:54.938 INFO:teuthology.orchestra.run.smithi061.stderr:real 0m0.163s 2023-12-25T21:15:54.938 INFO:teuthology.orchestra.run.smithi061.stderr:user 0m0.240s 2023-12-25T21:15:54.938 INFO:teuthology.orchestra.run.smithi061.stderr:sys 0m0.017s 2023-12-25T21:15:54.938 INFO:tasks.cephadm:Archiving logs... 2023-12-25T21:15:54.939 DEBUG:teuthology.misc:Transferring archived files from smithi061:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500977/remote/ubuntu@smithi061.front.sepia.ceph.com/log 2023-12-25T21:15:54.939 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-25T21:15:55.061 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500977/remote/ubuntu@smithi174.front.sepia.ceph.com/log 2023-12-25T21:15:55.062 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-25T21:15:55.089 INFO:tasks.cephadm:Removing cluster... 2023-12-25T21:15:55.090 DEBUG:teuthology.orchestra.run.smithi061:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5532d712-a369-11ee-95a5-87774f69a715 --force 2023-12-25T21:16:52.024 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5532d712-a369-11ee-95a5-87774f69a715 --force 2023-12-25T21:16:52.330 INFO:tasks.cephadm:Removing cephadm ... 2023-12-25T21:16:52.330 DEBUG:teuthology.orchestra.run.smithi061:> rm -rf /home/ubuntu/cephtest/cephadm 2023-12-25T21:16:52.350 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf /home/ubuntu/cephtest/cephadm 2023-12-25T21:16:52.368 INFO:tasks.cephadm:Teardown complete 2023-12-25T21:16:52.368 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi174 with status 5: 'sudo systemctl stop ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi174' 2023-12-25T21:16:52.454 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=74cee0e9b865474cacde0118523bbda0 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b14a0a106b8cffae0144ff9fb83d9ce1af1a4bd2/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi174 with status 5: 'sudo systemctl stop ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi174' 2023-12-25T21:16:52.458 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-12-25T21:16:52.467 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-25T21:16:52.485 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-25T21:16:52.485 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-25T21:16:52.504 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-25T21:16:52.577 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-25T21:16:52.587 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-25T21:16:52.587 DEBUG:teuthology.orchestra.run.smithi061:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-25T21:16:52.590 DEBUG:teuthology.orchestra.run.smithi174:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-25T21:16:52.606 INFO:teuthology.orchestra.run.smithi061.stderr:bash: ntpq: command not found 2023-12-25T21:16:52.607 INFO:teuthology.orchestra.run.smithi174.stderr:bash: ntpq: command not found 2023-12-25T21:16:52.610 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-25T21:16:52.610 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2023-12-25T21:16:52.610 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 5 +286us[ +273us] +/- 73ms 2023-12-25T21:16:52.610 INFO:teuthology.orchestra.run.smithi174.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 -70us[ -84us] +/- 58ms 2023-12-25T21:16:52.611 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 -168us[ -182us] +/- 63ms 2023-12-25T21:16:52.611 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T21:16:52.612 INFO:teuthology.orchestra.run.smithi061.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-25T21:16:52.612 INFO:teuthology.orchestra.run.smithi061.stdout:=============================================================================== 2023-12-25T21:16:52.612 INFO:teuthology.orchestra.run.smithi061.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 3 +295us[ +295us] +/- 73ms 2023-12-25T21:16:52.612 INFO:teuthology.orchestra.run.smithi061.stdout:^* hv02.front.sepia.ceph.com 3 6 377 3 -55us[ -41us] +/- 58ms 2023-12-25T21:16:52.613 INFO:teuthology.orchestra.run.smithi061.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 -136us[ -122us] +/- 63ms 2023-12-25T21:16:52.613 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-25T21:16:52.613 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-25T21:16:52.623 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-25T21:16:52.624 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-25T21:16:52.633 DEBUG:teuthology.orchestra.run.smithi061:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-25T21:16:52.744 DEBUG:teuthology.orchestra.run.smithi174:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-25T21:16:52.835 DEBUG:teuthology.orchestra.run.smithi061:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-25T21:16:52.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T21:16:52.868 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-25T21:16:52.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T21:16:52.901 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-25T21:16:52.929 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-25T21:16:52.966 INFO:teuthology.task.internal:Duration was 1381.462988 seconds 2023-12-25T21:16:52.966 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-25T21:16:52.976 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-25T21:16:52.976 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-25T21:16:52.978 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-25T21:16:53.023 INFO:teuthology.orchestra.run.smithi061.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-25T21:16:53.024 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-25T21:16:53.215 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-25T21:16:53.216 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi061.front.sepia.ceph.com 2023-12-25T21:16:53.216 DEBUG:teuthology.orchestra.run.smithi061:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-25T21:16:53.240 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi174.front.sepia.ceph.com 2023-12-25T21:16:53.240 DEBUG:teuthology.orchestra.run.smithi174:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-25T21:16:53.263 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-25T21:16:53.263 DEBUG:teuthology.orchestra.run.smithi061:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-25T21:16:53.282 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-25T21:16:53.349 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-25T21:16:53.349 DEBUG:teuthology.orchestra.run.smithi061:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-25T21:16:53.351 DEBUG:teuthology.orchestra.run.smithi174:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-25T21:16:53.437 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-25T21:16:53.448 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-25T21:16:53.448 DEBUG:teuthology.orchestra.run.smithi061:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-25T21:16:53.476 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-25T21:16:53.503 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-25T21:16:53.516 DEBUG:teuthology.orchestra.run.smithi061:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-25T21:16:53.519 DEBUG:teuthology.orchestra.run.smithi174:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-25T21:16:53.544 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern = core 2023-12-25T21:16:53.570 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = core 2023-12-25T21:16:53.636 DEBUG:teuthology.orchestra.run.smithi061:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-25T21:16:53.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T21:16:53.654 DEBUG:teuthology.orchestra.run.smithi174:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-25T21:16:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-25T21:16:53.693 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-25T21:16:53.704 INFO:teuthology.task.internal:Transferring archived files... 2023-12-25T21:16:53.704 DEBUG:teuthology.misc:Transferring archived files from smithi061:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500977/remote/smithi061 2023-12-25T21:16:53.705 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-25T21:16:53.745 DEBUG:teuthology.misc:Transferring archived files from smithi174:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500977/remote/smithi174 2023-12-25T21:16:53.746 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-25T21:16:53.785 INFO:teuthology.task.internal:Removing archive directory... 2023-12-25T21:16:53.786 DEBUG:teuthology.orchestra.run.smithi061:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-25T21:16:53.788 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-25T21:16:53.838 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-25T21:16:53.851 INFO:teuthology.task.internal:Not uploading archives. 2023-12-25T21:16:53.851 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-25T21:16:53.940 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-25T21:16:53.941 DEBUG:teuthology.orchestra.run.smithi061:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-25T21:16:53.943 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-25T21:16:53.959 INFO:teuthology.orchestra.run.smithi061.stdout: 525952 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 25 21:16 /home/ubuntu/cephtest 2023-12-25T21:16:53.960 INFO:teuthology.orchestra.run.smithi174.stdout: 525359 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 25 21:16 /home/ubuntu/cephtest 2023-12-25T21:16:53.961 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-25T21:16:53.975 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-25T21:16:54.007 INFO:teuthology.nuke:Checking targets against current locks 2023-12-25T21:16:54.032 DEBUG:teuthology.nuke:shortname: smithi061 2023-12-25T21:16:54.032 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-25T21:16:54.058 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi061.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500977', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-25 20:45:16.868182', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNGWUdfyqeJ/EhV66Hng+Yju49pNNiEPvGvBkWe82OrfjGFxgf42CPBREXpFsKHIDf9ZSodY2pEzA9nyOE3h6KI='} 2023-12-25T21:16:54.059 DEBUG:teuthology.nuke:shortname: smithi174 2023-12-25T21:16:54.060 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-25T21:16:54.083 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-25_17:28:36-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7500977', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-25 20:45:16.866965', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBXyRFTdwGQLNHrZkx91RjrtdgcYVey/TBmxRFQkadX/e83vU7kXA9bqPckj6BeBlGlKQxXF7BVv32LD9zoZ8ig='} 2023-12-25T21:16:54.117 INFO:teuthology.orchestra.console.smithi061:Power off 2023-12-25T21:16:54.118 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-25T21:16:54.141 DEBUG:teuthology.orchestra.console.smithi061:power off output: Chassis Power Control: Down/Off 2023-12-25T21:16:54.142 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T21:16:54.159 INFO:teuthology.orchestra.console.smithi174:Power off 2023-12-25T21:16:54.160 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-25T21:16:54.170 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is on 2023-12-25T21:16:54.182 DEBUG:teuthology.orchestra.console.smithi174:power off output: Chassis Power Control: Down/Off 2023-12-25T21:16:54.183 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T21:16:54.206 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2023-12-25T21:16:58.171 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T21:16:58.207 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T21:16:58.282 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is on 2023-12-25T21:16:58.317 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2023-12-25T21:17:02.287 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T21:17:02.318 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-25T21:17:02.398 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is off 2023-12-25T21:17:02.429 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is off 2023-12-25T21:17:02.500 INFO:teuthology.orchestra.console.smithi061:Power off completed 2023-12-25T21:17:02.530 INFO:teuthology.orchestra.console.smithi174:Power off completed 2023-12-25T21:17:02.635 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1381.4629881381989 failure_reason: 'Command failed on smithi174 with status 5: ''sudo systemctl stop ceph-5532d712-a369-11ee-95a5-87774f69a715@mon.smithi174''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=74cee0e9b865474cacde0118523bbda0 status: fail success: false 2023-12-25T21:17:02.636 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-25T21:17:02.717 INFO:teuthology.run:FAIL