2024-01-13T11:34:41.876 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-13T11:34:41.876 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --name yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516050 -- /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516050/orig.config.yaml 2024-01-13T11:34:41.925 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-13T11:34:42.015 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516050 branch: wip-yuri2-testing-2024-01-12-1128-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7516050' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri2-testing-2024-01-12-1128-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 445562ab4bc3ddfb386936119050695810860bcb ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 445562ab4bc3ddfb386936119050695810860bcb selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri2-testing-2024-01-12-1128-pacific sha1: 445562ab4bc3ddfb386936119050695810860bcb owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 5246 sha1: 445562ab4bc3ddfb386936119050695810860bcb sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri2-testing-2024-01-12-1128-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 445562ab4bc3ddfb386936119050695810860bcb targets: smithi059.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLe/qubIVW8mPTHtv8LfzeNUpJlwG55ekiV/OKHpQSrMAuXJfj3B67+ixTM7TOPMnqW5mNj0YIBWLlH9Pu46Ds0= smithi123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIIIhmroZg0ERs05ifmQozDoK5B7BmGFr/SA5Tn+H2Xs2WcmrNNhpZY2zOiY28R1ZginFOa+cPP3jrsRA6fVz28= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-12_21:52:25 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-01-13T11:34:42.015 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa; will attempt to use it 2024-01-13T11:34:42.016 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks 2024-01-13T11:34:42.016 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-13T11:34:42.018 INFO:teuthology.task.internal:Checking packages... 2024-01-13T11:34:42.041 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '445562ab4bc3ddfb386936119050695810860bcb' 2024-01-13T11:34:42.041 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-13T11:34:42.042 INFO:teuthology.packaging:ref: None 2024-01-13T11:34:42.042 INFO:teuthology.packaging:tag: None 2024-01-13T11:34:42.042 INFO:teuthology.packaging:branch: wip-yuri2-testing-2024-01-12-1128-pacific 2024-01-13T11:34:42.042 INFO:teuthology.packaging:sha1: 445562ab4bc3ddfb386936119050695810860bcb 2024-01-13T11:34:42.042 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri2-testing-2024-01-12-1128-pacific 2024-01-13T11:34:42.256 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-456.g445562ab 2024-01-13T11:34:42.258 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-13T11:34:42.265 INFO:teuthology.task.internal:no buildpackages task found 2024-01-13T11:34:42.265 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-13T11:34:42.299 INFO:teuthology.task.internal:Saving configuration 2024-01-13T11:34:42.317 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-13T11:34:42.323 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-13T11:34:42.348 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516050', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 11:29:34.001224', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLe/qubIVW8mPTHtv8LfzeNUpJlwG55ekiV/OKHpQSrMAuXJfj3B67+ixTM7TOPMnqW5mNj0YIBWLlH9Pu46Ds0='} 2024-01-13T11:34:42.371 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516050', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 11:29:34.002840', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIIIhmroZg0ERs05ifmQozDoK5B7BmGFr/SA5Tn+H2Xs2WcmrNNhpZY2zOiY28R1ZginFOa+cPP3jrsRA6fVz28='} 2024-01-13T11:34:42.372 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-13T11:34:42.379 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-01-13T11:34:42.380 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-01-13T11:34:42.380 INFO:teuthology.run_tasks:Running task console_log... 2024-01-13T11:34:42.447 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3312c07160>, signals=[15]) 2024-01-13T11:34:42.447 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-13T11:34:42.453 INFO:teuthology.task.internal:Opening connections... 2024-01-13T11:34:42.454 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2024-01-13T11:34:42.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:34:42.534 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2024-01-13T11:34:42.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:34:42.608 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-13T11:34:42.614 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2024-01-13T11:34:42.637 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2024-01-13T11:34:42.637 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2024-01-13T11:34:42.699 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="CentOS Stream" 2024-01-13T11:34:42.699 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="8" 2024-01-13T11:34:42.699 INFO:teuthology.orchestra.run.smithi059.stdout:ID="centos" 2024-01-13T11:34:42.700 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE="rhel fedora" 2024-01-13T11:34:42.700 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="8" 2024-01-13T11:34:42.700 INFO:teuthology.orchestra.run.smithi059.stdout:PLATFORM_ID="platform:el8" 2024-01-13T11:34:42.700 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-13T11:34:42.700 INFO:teuthology.orchestra.run.smithi059.stdout:ANSI_COLOR="0;31" 2024-01-13T11:34:42.700 INFO:teuthology.orchestra.run.smithi059.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-13T11:34:42.700 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://centos.org/" 2024-01-13T11:34:42.700 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-13T11:34:42.700 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-13T11:34:42.700 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-13T11:34:42.701 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2024-01-13T11:34:42.727 DEBUG:teuthology.orchestra.run.smithi123:> uname -m 2024-01-13T11:34:42.752 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2024-01-13T11:34:42.752 DEBUG:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2024-01-13T11:34:42.813 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="CentOS Stream" 2024-01-13T11:34:42.813 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="8" 2024-01-13T11:34:42.813 INFO:teuthology.orchestra.run.smithi123.stdout:ID="centos" 2024-01-13T11:34:42.813 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE="rhel fedora" 2024-01-13T11:34:42.813 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="8" 2024-01-13T11:34:42.814 INFO:teuthology.orchestra.run.smithi123.stdout:PLATFORM_ID="platform:el8" 2024-01-13T11:34:42.814 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-13T11:34:42.814 INFO:teuthology.orchestra.run.smithi123.stdout:ANSI_COLOR="0;31" 2024-01-13T11:34:42.814 INFO:teuthology.orchestra.run.smithi123.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-13T11:34:42.814 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://centos.org/" 2024-01-13T11:34:42.814 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-13T11:34:42.814 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-13T11:34:42.814 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-13T11:34:42.814 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2024-01-13T11:34:42.835 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-13T11:34:42.844 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-13T11:34:42.851 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-13T11:34:42.851 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2024-01-13T11:34:42.854 DEBUG:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2024-01-13T11:34:42.875 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-13T11:34:42.881 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-13T11:34:42.881 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2024-01-13T11:34:42.916 DEBUG:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2024-01-13T11:34:42.946 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-13T11:34:42.996 INFO:teuthology.run_tasks:Running task kernel... 2024-01-13T11:34:43.011 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-13T11:34:43.011 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-13T11:34:43.011 DEBUG:teuthology.orchestra.run.smithi059:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-13T11:34:43.012 DEBUG:teuthology.orchestra.run.smithi123:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-13T11:34:43.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:34:43.033 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-01-13T11:34:43.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:34:43.034 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-01-13T11:34:43.095 INFO:teuthology.orchestra.run.smithi123.stdout:4.18.0-489.el8.x86_64 2024-01-13T11:34:43.095 INFO:teuthology.task.kernel:Running kernel on smithi123: 4.18.0-489.el8.x86_64 2024-01-13T11:34:43.096 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum install -y kernel 2024-01-13T11:34:43.097 INFO:teuthology.orchestra.run.smithi059.stdout:4.18.0-489.el8.x86_64 2024-01-13T11:34:43.097 INFO:teuthology.task.kernel:Running kernel on smithi059: 4.18.0-489.el8.x86_64 2024-01-13T11:34:43.097 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum install -y kernel 2024-01-13T11:34:47.196 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - AppStream 23 MB/s | 35 MB 00:01 2024-01-13T11:34:47.287 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - AppStream 21 MB/s | 35 MB 00:01 2024-01-13T11:34:55.287 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - BaseOS 33 MB/s | 58 MB 00:01 2024-01-13T11:34:57.154 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - BaseOS 19 MB/s | 58 MB 00:03 2024-01-13T11:35:03.728 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras 75 kB/s | 18 kB 00:00 2024-01-13T11:35:04.287 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2024-01-13T11:35:04.581 INFO:teuthology.orchestra.run.smithi123.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-01-13T11:35:06.153 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - Extras 86 kB/s | 18 kB 00:00 2024-01-13T11:35:06.748 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2024-01-13T11:35:07.155 INFO:teuthology.orchestra.run.smithi059.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2024-01-13T11:35:08.114 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 4.8 MB/s | 16 MB 00:03 2024-01-13T11:35:11.980 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 386 kB/s | 24 kB 00:00 2024-01-13T11:35:12.317 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 3.2 MB/s | 16 MB 00:04 2024-01-13T11:35:15.739 INFO:teuthology.orchestra.run.smithi123.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-13T11:35:15.739 INFO:teuthology.orchestra.run.smithi123.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-13T11:35:16.041 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:35:16.041 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:35:16.042 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-01-13T11:35:16.042 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:35:16.042 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-01-13T11:35:16.042 INFO:teuthology.orchestra.run.smithi123.stdout: kernel x86_64 4.18.0-532.el8 baseos 10 M 2024-01-13T11:35:16.042 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-01-13T11:35:16.042 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-core x86_64 4.18.0-532.el8 baseos 43 M 2024-01-13T11:35:16.042 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules x86_64 4.18.0-532.el8 baseos 36 M 2024-01-13T11:35:16.042 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:35:16.042 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-13T11:35:16.042 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:35:16.042 INFO:teuthology.orchestra.run.smithi123.stdout:Install 3 Packages 2024-01-13T11:35:16.042 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:35:16.043 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 90 M 2024-01-13T11:35:16.043 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 96 M 2024-01-13T11:35:16.043 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-01-13T11:35:16.397 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 460 kB/s | 24 kB 00:00 2024-01-13T11:35:18.548 INFO:teuthology.orchestra.run.smithi123.stdout:(1/3): kernel-4.18.0-532.el8.x86_64.rpm 4.4 MB/s | 10 MB 00:02 2024-01-13T11:35:19.298 INFO:teuthology.orchestra.run.smithi123.stdout:(2/3): kernel-core-4.18.0-532.el8.x86_64.rpm 14 MB/s | 43 MB 00:03 2024-01-13T11:35:20.255 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-13T11:35:20.255 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-13T11:35:20.577 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:35:20.578 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:35:20.578 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-01-13T11:35:20.578 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:35:20.579 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-13T11:35:20.579 INFO:teuthology.orchestra.run.smithi059.stdout: kernel x86_64 4.18.0-532.el8 baseos 10 M 2024-01-13T11:35:20.579 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-01-13T11:35:20.579 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core x86_64 4.18.0-532.el8 baseos 43 M 2024-01-13T11:35:20.579 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules x86_64 4.18.0-532.el8 baseos 36 M 2024-01-13T11:35:20.579 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:35:20.579 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-13T11:35:20.579 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:35:20.579 INFO:teuthology.orchestra.run.smithi059.stdout:Install 3 Packages 2024-01-13T11:35:20.579 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:35:20.580 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 90 M 2024-01-13T11:35:20.580 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 96 M 2024-01-13T11:35:20.580 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-13T11:35:20.606 INFO:teuthology.orchestra.run.smithi123.stdout:(3/3): kernel-modules-4.18.0-532.el8.x86_64.rpm 8.3 MB/s | 36 MB 00:04 2024-01-13T11:35:20.607 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:35:20.607 INFO:teuthology.orchestra.run.smithi123.stdout:Total 20 MB/s | 90 MB 00:04 2024-01-13T11:35:21.289 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-01-13T11:35:21.573 INFO:teuthology.orchestra.run.smithi059.stdout:(1/3): kernel-4.18.0-532.el8.x86_64.rpm 13 MB/s | 10 MB 00:00 2024-01-13T11:35:21.703 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-01-13T11:35:21.703 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-01-13T11:35:22.740 INFO:teuthology.orchestra.run.smithi059.stdout:(2/3): kernel-modules-4.18.0-532.el8.x86_64.rpm 18 MB/s | 36 MB 00:01 2024-01-13T11:35:23.142 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-01-13T11:35:23.168 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-01-13T11:35:24.074 INFO:teuthology.orchestra.run.smithi059.stdout:(3/3): kernel-core-4.18.0-532.el8.x86_64.rpm 13 MB/s | 43 MB 00:03 2024-01-13T11:35:24.074 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:35:24.074 INFO:teuthology.orchestra.run.smithi059.stdout:Total 26 MB/s | 90 MB 00:03 2024-01-13T11:35:24.784 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-13T11:35:25.229 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-13T11:35:25.229 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-13T11:35:25.966 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-01-13T11:35:26.530 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-13T11:35:26.538 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-core-4.18.0-532.el8.x86_64 1/3 2024-01-13T11:35:26.650 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-13T11:35:29.604 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-13T11:35:30.082 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-core-4.18.0-532.el8.x86_64 1/3 2024-01-13T11:35:30.108 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-core-4.18.0-532.el8.x86_64 1/3 2024-01-13T11:35:30.504 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-modules-4.18.0-532.el8.x86_64 2/3 2024-01-13T11:35:33.871 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-4.18.0-532.el8.x86_64 1/3 2024-01-13T11:35:33.941 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-modules-4.18.0-532.el8.x86_64 2/3 2024-01-13T11:35:34.235 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-4.18.0-532.el8.x86_64 3/3 2024-01-13T11:35:34.274 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-4.18.0-532.el8.x86_64 2/3 2024-01-13T11:35:37.950 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-4.18.0-532.el8.x86_64 2/3 2024-01-13T11:35:38.227 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-4.18.0-532.el8.x86_64 3/3 2024-01-13T11:35:56.058 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-core-4.18.0-532.el8.x86_64 3/3 2024-01-13T11:35:56.080 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-modules-4.18.0-532.el8.x86_64 3/3 2024-01-13T11:35:56.415 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-4.18.0-532.el8.x86_64 3/3 2024-01-13T11:35:56.416 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-4.18.0-532.el8.x86_64 1/3 2024-01-13T11:35:56.416 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-core-4.18.0-532.el8.x86_64 2/3 2024-01-13T11:35:57.208 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-modules-4.18.0-532.el8.x86_64 3/3 2024-01-13T11:35:57.208 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:35:57.208 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-01-13T11:35:57.208 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-4.18.0-532.el8.x86_64 kernel-core-4.18.0-532.el8.x86_64 2024-01-13T11:35:57.208 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules-4.18.0-532.el8.x86_64 2024-01-13T11:35:57.208 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:35:57.208 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:35:57.362 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q kernel | sort -rV | head -n 1 2024-01-13T11:35:57.579 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-4.18.0-532.el8.x86_64 2024-01-13T11:35:57.579 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-532.el8.x86_64 2024-01-13T11:35:57.580 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-532.el8.x86_64 2024-01-13T11:35:57.580 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-13T11:35:57.580 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-13T11:35:57.581 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-01-13T11:35:57.581 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi123.front.sepia.ceph.com, path=None, version=distro) 2024-01-13T11:35:57.581 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q kernel | sort -rV | head -n 1 2024-01-13T11:35:57.791 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-4.18.0-532.el8.x86_64 2024-01-13T11:35:57.791 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-532.el8.x86_64 2024-01-13T11:35:57.791 DEBUG:teuthology.orchestra.run.smithi123:> sudo rpm -qi grub2-tools 2024-01-13T11:35:57.825 INFO:teuthology.orchestra.run.smithi123.stdout:Name : grub2-tools 2024-01-13T11:35:57.825 INFO:teuthology.orchestra.run.smithi123.stdout:Epoch : 1 2024-01-13T11:35:57.825 INFO:teuthology.orchestra.run.smithi123.stdout:Version : 2.02 2024-01-13T11:35:57.825 INFO:teuthology.orchestra.run.smithi123.stdout:Release : 129.el8 2024-01-13T11:35:57.825 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: x86_64 2024-01-13T11:35:57.826 INFO:teuthology.orchestra.run.smithi123.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-13T11:35:57.826 INFO:teuthology.orchestra.run.smithi123.stdout:Group : System Environment/Base 2024-01-13T11:35:57.826 INFO:teuthology.orchestra.run.smithi123.stdout:Size : 9265586 2024-01-13T11:35:57.826 INFO:teuthology.orchestra.run.smithi123.stdout:License : GPLv3+ 2024-01-13T11:35:57.826 INFO:teuthology.orchestra.run.smithi123.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-13T11:35:57.826 INFO:teuthology.orchestra.run.smithi123.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-13T11:35:57.826 INFO:teuthology.orchestra.run.smithi123.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-13T11:35:57.826 INFO:teuthology.orchestra.run.smithi123.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-13T11:35:57.826 INFO:teuthology.orchestra.run.smithi123.stdout:Relocations : (not relocatable) 2024-01-13T11:35:57.826 INFO:teuthology.orchestra.run.smithi123.stdout:Packager : CentOS BuildSystem 2024-01-13T11:35:57.826 INFO:teuthology.orchestra.run.smithi123.stdout:Vendor : CentOS 2024-01-13T11:35:57.826 INFO:teuthology.orchestra.run.smithi123.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-13T11:35:57.826 INFO:teuthology.orchestra.run.smithi123.stdout:Summary : Support tools for GRUB. 2024-01-13T11:35:57.826 INFO:teuthology.orchestra.run.smithi123.stdout:Description : 2024-01-13T11:35:57.827 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:35:57.827 INFO:teuthology.orchestra.run.smithi123.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-13T11:35:57.827 INFO:teuthology.orchestra.run.smithi123.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-13T11:35:57.827 INFO:teuthology.orchestra.run.smithi123.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-13T11:35:57.827 INFO:teuthology.orchestra.run.smithi123.stdout:hardware devices. 2024-01-13T11:35:57.827 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:35:57.827 INFO:teuthology.orchestra.run.smithi123.stdout:This subpackage provides tools for support of all platforms. 2024-01-13T11:35:57.828 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-13T11:35:57.828 INFO:teuthology.task.kernel:Updating grub on smithi123 to boot 4.18.0-532.el8.x86_64 2024-01-13T11:35:57.828 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-13T11:35:59.502 INFO:teuthology.orchestra.run.smithi123.stderr:Generating grub configuration file ... 2024-01-13T11:36:01.660 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-4.18.0-532.el8.x86_64 3/3 2024-01-13T11:36:01.683 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-4.18.0-532.el8.x86_64 3/3 2024-01-13T11:36:01.830 INFO:teuthology.orchestra.run.smithi123.stderr:done 2024-01-13T11:36:01.831 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2024-01-13T11:36:01.875 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.FvWUVNLeX1 2024-01-13T11:36:01.875 DEBUG:teuthology.orchestra.run.smithi123:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.FvWUVNLeX1 2024-01-13T11:36:01.910 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.FvWUVNLeX1 2024-01-13T11:36:02.045 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-4.18.0-532.el8.x86_64 3/3 2024-01-13T11:36:02.045 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-4.18.0-532.el8.x86_64 1/3 2024-01-13T11:36:02.045 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-core-4.18.0-532.el8.x86_64 2/3 2024-01-13T11:36:02.071 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.FvWUVNLeX1 is 6KB 2024-01-13T11:36:02.119 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.FvWUVNLeX1 2024-01-13T11:36:02.137 DEBUG:teuthology.orchestra.run.smithi123:> sudo /bin/ls /boot/loader/entries || true 2024-01-13T11:36:02.202 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-13T11:36:02.202 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-13T11:36:02.202 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-13T11:36:02.202 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-532.el8.x86_64.conf 2024-01-13T11:36:02.203 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-532.el8.x86_64 2024-01-13T11:36:02.389 DEBUG:teuthology.orchestra.run.smithi123:> sudo shutdown -r now 2024-01-13T11:36:02.876 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-4.18.0-532.el8.x86_64 3/3 2024-01-13T11:36:02.876 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:36:02.876 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-13T11:36:02.876 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-4.18.0-532.el8.x86_64 kernel-core-4.18.0-532.el8.x86_64 2024-01-13T11:36:02.876 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-4.18.0-532.el8.x86_64 2024-01-13T11:36:02.876 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:36:02.876 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:36:03.035 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2024-01-13T11:36:03.258 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-4.18.0-532.el8.x86_64 2024-01-13T11:36:03.258 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-532.el8.x86_64 2024-01-13T11:36:03.259 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-532.el8.x86_64 2024-01-13T11:36:03.259 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-13T11:36:03.259 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-13T11:36:03.259 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-01-13T11:36:03.259 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi059.front.sepia.ceph.com, path=None, version=distro) 2024-01-13T11:36:03.259 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2024-01-13T11:36:03.479 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-4.18.0-532.el8.x86_64 2024-01-13T11:36:03.479 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-532.el8.x86_64 2024-01-13T11:36:03.479 DEBUG:teuthology.orchestra.run.smithi059:> sudo rpm -qi grub2-tools 2024-01-13T11:36:03.513 INFO:teuthology.orchestra.run.smithi059.stdout:Name : grub2-tools 2024-01-13T11:36:03.513 INFO:teuthology.orchestra.run.smithi059.stdout:Epoch : 1 2024-01-13T11:36:03.514 INFO:teuthology.orchestra.run.smithi059.stdout:Version : 2.02 2024-01-13T11:36:03.514 INFO:teuthology.orchestra.run.smithi059.stdout:Release : 129.el8 2024-01-13T11:36:03.514 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: x86_64 2024-01-13T11:36:03.514 INFO:teuthology.orchestra.run.smithi059.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-13T11:36:03.514 INFO:teuthology.orchestra.run.smithi059.stdout:Group : System Environment/Base 2024-01-13T11:36:03.514 INFO:teuthology.orchestra.run.smithi059.stdout:Size : 9265586 2024-01-13T11:36:03.514 INFO:teuthology.orchestra.run.smithi059.stdout:License : GPLv3+ 2024-01-13T11:36:03.514 INFO:teuthology.orchestra.run.smithi059.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-13T11:36:03.514 INFO:teuthology.orchestra.run.smithi059.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-13T11:36:03.514 INFO:teuthology.orchestra.run.smithi059.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-13T11:36:03.514 INFO:teuthology.orchestra.run.smithi059.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-13T11:36:03.514 INFO:teuthology.orchestra.run.smithi059.stdout:Relocations : (not relocatable) 2024-01-13T11:36:03.514 INFO:teuthology.orchestra.run.smithi059.stdout:Packager : CentOS BuildSystem 2024-01-13T11:36:03.515 INFO:teuthology.orchestra.run.smithi059.stdout:Vendor : CentOS 2024-01-13T11:36:03.515 INFO:teuthology.orchestra.run.smithi059.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-13T11:36:03.515 INFO:teuthology.orchestra.run.smithi059.stdout:Summary : Support tools for GRUB. 2024-01-13T11:36:03.515 INFO:teuthology.orchestra.run.smithi059.stdout:Description : 2024-01-13T11:36:03.515 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:36:03.515 INFO:teuthology.orchestra.run.smithi059.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-13T11:36:03.515 INFO:teuthology.orchestra.run.smithi059.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-13T11:36:03.515 INFO:teuthology.orchestra.run.smithi059.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-13T11:36:03.515 INFO:teuthology.orchestra.run.smithi059.stdout:hardware devices. 2024-01-13T11:36:03.515 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:36:03.515 INFO:teuthology.orchestra.run.smithi059.stdout:This subpackage provides tools for support of all platforms. 2024-01-13T11:36:03.516 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-13T11:36:03.516 INFO:teuthology.task.kernel:Updating grub on smithi059 to boot 4.18.0-532.el8.x86_64 2024-01-13T11:36:03.516 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-13T11:36:04.929 INFO:teuthology.orchestra.run.smithi059.stderr:Generating grub configuration file ... 2024-01-13T11:36:07.044 INFO:teuthology.orchestra.run.smithi059.stderr:done 2024-01-13T11:36:07.046 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-01-13T11:36:07.075 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.QDvJdWvGFN 2024-01-13T11:36:07.075 DEBUG:teuthology.orchestra.run.smithi059:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QDvJdWvGFN 2024-01-13T11:36:07.147 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.QDvJdWvGFN 2024-01-13T11:36:07.291 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.QDvJdWvGFN is 6KB 2024-01-13T11:36:07.339 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.QDvJdWvGFN 2024-01-13T11:36:07.357 DEBUG:teuthology.orchestra.run.smithi059:> sudo /bin/ls /boot/loader/entries || true 2024-01-13T11:36:07.431 INFO:teuthology.orchestra.run.smithi059.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-13T11:36:07.431 INFO:teuthology.orchestra.run.smithi059.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-13T11:36:07.432 INFO:teuthology.orchestra.run.smithi059.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-13T11:36:07.432 INFO:teuthology.orchestra.run.smithi059.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-532.el8.x86_64.conf 2024-01-13T11:36:07.433 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-532.el8.x86_64 2024-01-13T11:36:07.592 DEBUG:teuthology.orchestra.run.smithi059:> sudo shutdown -r now 2024-01-13T11:36:32.402 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-01-13T11:36:32.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-01-13T11:36:32.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:36:37.600 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-01-13T11:36:37.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-01-13T11:36:37.600 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:36:50.748 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2024-01-13T11:36:56.128 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.59 2024-01-13T11:36:59.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-01-13T11:36:59.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:37:02.812 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2024-01-13T11:37:05.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-01-13T11:37:05.132 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:37:14.844 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-01-13T11:37:14.844 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:37:33.244 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2024-01-13T11:37:48.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-01-13T11:37:48.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:37:49.011 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-01-13T11:37:50.023 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-01-13T11:37:50.023 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-532.el8.x86_64"... 2024-01-13T11:37:50.024 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-01-13T11:37:50.040 INFO:teuthology.orchestra.run.smithi123.stdout:4.18.0-532.el8.x86_64 2024-01-13T11:37:50.041 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-532.el8.x86_64 vs 4.18.0-532.el8.x86_64 2024-01-13T11:37:50.041 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-13T11:37:50.041 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-13T11:37:51.042 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-13T11:37:51.042 DEBUG:teuthology.orchestra.run.smithi123:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-13T11:37:51.118 INFO:teuthology.orchestra.run.smithi123.stdout:ttyS1 2024-01-13T11:37:51.134 DEBUG:teuthology.parallel:result is None 2024-01-13T11:38:05.134 DEBUG:teuthology.orchestra.remote:timed out 2024-01-13T11:38:17.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-01-13T11:38:17.135 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:38:17.457 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-01-13T11:38:17.784 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-01-13T11:38:17.785 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-532.el8.x86_64"... 2024-01-13T11:38:17.785 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-01-13T11:38:17.806 INFO:teuthology.orchestra.run.smithi059.stdout:4.18.0-532.el8.x86_64 2024-01-13T11:38:17.806 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-532.el8.x86_64 vs 4.18.0-532.el8.x86_64 2024-01-13T11:38:17.806 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-13T11:38:17.806 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-13T11:38:18.807 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-13T11:38:18.807 DEBUG:teuthology.orchestra.run.smithi059:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-13T11:38:18.905 INFO:teuthology.orchestra.run.smithi059.stdout:ttyS1 2024-01-13T11:38:18.918 DEBUG:teuthology.parallel:result is None 2024-01-13T11:38:18.918 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-13T11:38:18.924 INFO:teuthology.task.internal:Creating test directory... 2024-01-13T11:38:18.924 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-13T11:38:18.927 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-13T11:38:18.949 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-13T11:38:18.955 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-13T11:38:18.964 INFO:teuthology.task.internal:Creating archive directory... 2024-01-13T11:38:18.964 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-13T11:38:18.988 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-13T11:38:19.028 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-13T11:38:19.035 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-13T11:38:19.035 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-13T11:38:19.066 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-13T11:38:19.101 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T11:38:19.103 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T11:38:19.112 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T11:38:19.116 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T11:38:19.117 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-13T11:38:19.124 INFO:teuthology.task.internal:Configuring sudo... 2024-01-13T11:38:19.124 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-13T11:38:19.156 DEBUG:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-13T11:38:19.191 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-13T11:38:19.201 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-13T11:38:19.201 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-13T11:38:19.228 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-13T11:38:19.253 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-13T11:38:19.334 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-13T11:38:19.382 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-13T11:38:19.469 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-13T11:38:19.512 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-13T11:38:19.512 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-13T11:38:19.580 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-13T11:38:19.627 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-13T11:38:19.690 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-13T11:38:19.738 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-13T11:38:19.787 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-01-13T11:38:19.787 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-13T11:38:19.861 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2024-01-13T11:38:19.863 DEBUG:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2024-01-13T11:38:19.926 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-13T11:38:19.975 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-13T11:38:20.196 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-13T11:38:20.203 INFO:teuthology.task.internal:Starting timer... 2024-01-13T11:38:20.203 INFO:teuthology.run_tasks:Running task pcp... 2024-01-13T11:38:20.225 INFO:teuthology.run_tasks:Running task selinux... 2024-01-13T11:38:20.233 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-13T11:38:20.233 DEBUG:teuthology.orchestra.run.smithi059:> sudo service auditd rotate 2024-01-13T11:38:20.319 INFO:teuthology.orchestra.run.smithi059.stdout:Rotating logs: 2024-01-13T11:38:20.321 DEBUG:teuthology.orchestra.run.smithi123:> sudo service auditd rotate 2024-01-13T11:38:20.422 INFO:teuthology.orchestra.run.smithi123.stdout:Rotating logs: 2024-01-13T11:38:20.423 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-13T11:38:20.424 DEBUG:teuthology.orchestra.run.smithi059:> /usr/sbin/getenforce 2024-01-13T11:38:20.452 INFO:teuthology.orchestra.run.smithi059.stdout:Permissive 2024-01-13T11:38:20.453 DEBUG:teuthology.orchestra.run.smithi123:> /usr/sbin/getenforce 2024-01-13T11:38:20.497 INFO:teuthology.orchestra.run.smithi123.stdout:Permissive 2024-01-13T11:38:20.497 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi059.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi123.front.sepia.ceph.com': 'permissive'} 2024-01-13T11:38:20.497 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-13T11:38:20.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:38:20.525 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-13T11:38:20.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:38:20.566 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-13T11:38:20.566 DEBUG:teuthology.orchestra.run.smithi059:> sudo /usr/sbin/setenforce permissive 2024-01-13T11:38:20.603 DEBUG:teuthology.orchestra.run.smithi123:> sudo /usr/sbin/setenforce permissive 2024-01-13T11:38:20.635 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-13T11:38:20.645 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-13T11:38:20.663 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-13T11:38:20.665 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi059.front.sepia.ceph.com,smithi123.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-13T11:44:20.168 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi059.front.sepia.ceph.com'), Remote(name='ubuntu@smithi123.front.sepia.ceph.com')] 2024-01-13T11:44:20.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-01-13T11:44:20.170 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:44:20.242 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-01-13T11:44:20.316 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-01-13T11:44:20.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-01-13T11:44:20.317 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:44:20.391 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-01-13T11:44:20.417 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-01-13T11:44:20.418 INFO:teuthology.run_tasks:Running task clock... 2024-01-13T11:44:20.427 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-13T11:44:20.427 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-13T11:44:20.428 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T11:44:20.430 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-13T11:44:20.430 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T11:44:20.462 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-13T11:44:20.477 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-13T11:44:20.489 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-13T11:44:20.503 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-13T11:44:20.541 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: ntpd: command not found 2024-01-13T11:44:20.552 INFO:teuthology.orchestra.run.smithi059.stdout:506 Cannot talk to daemon 2024-01-13T11:44:20.566 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-13T11:44:20.566 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: ntpd: command not found 2024-01-13T11:44:20.577 INFO:teuthology.orchestra.run.smithi123.stdout:506 Cannot talk to daemon 2024-01-13T11:44:20.579 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-13T11:44:20.590 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-13T11:44:20.603 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-13T11:44:20.645 INFO:teuthology.orchestra.run.smithi059.stderr:bash: ntpq: command not found 2024-01-13T11:44:20.648 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-13T11:44:20.649 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2024-01-13T11:44:20.649 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T11:44:20.649 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T11:44:20.649 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T11:44:20.649 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T11:44:20.670 INFO:teuthology.orchestra.run.smithi123.stderr:bash: ntpq: command not found 2024-01-13T11:44:20.674 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-13T11:44:20.674 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-01-13T11:44:20.674 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T11:44:20.674 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T11:44:20.674 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T11:44:20.674 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T11:44:20.675 INFO:teuthology.run_tasks:Running task pexec... 2024-01-13T11:44:20.684 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-13T11:44:20.685 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi059.front.sepia.ceph.com 2024-01-13T11:44:20.685 DEBUG:teuthology.orchestra.run.smithi059:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-13T11:44:20.685 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi123.front.sepia.ceph.com 2024-01-13T11:44:20.685 DEBUG:teuthology.orchestra.run.smithi123:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-13T11:44:21.602 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:40 ago on Sat 13 Jan 2024 11:42:41 AM UTC. 2024-01-13T11:44:21.613 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:33 ago on Sat 13 Jan 2024 11:42:48 AM UTC. 2024-01-13T11:44:23.802 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:44:23.803 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:44:23.803 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-01-13T11:44:23.803 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:44:23.803 INFO:teuthology.orchestra.run.smithi123.stdout:Resetting modules: 2024-01-13T11:44:23.803 INFO:teuthology.orchestra.run.smithi123.stdout: container-tools 2024-01-13T11:44:23.803 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:44:23.803 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-13T11:44:23.803 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:44:23.803 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:44:23.946 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:44:23.947 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:44:23.947 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-01-13T11:44:23.947 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:44:23.947 INFO:teuthology.orchestra.run.smithi059.stdout:Resetting modules: 2024-01-13T11:44:23.947 INFO:teuthology.orchestra.run.smithi059.stdout: container-tools 2024-01-13T11:44:23.947 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:44:23.947 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-13T11:44:23.947 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:44:23.947 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:44:24.043 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:44:24.206 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:44:24.982 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:43 ago on Sat 13 Jan 2024 11:42:41 AM UTC. 2024-01-13T11:44:25.184 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:37 ago on Sat 13 Jan 2024 11:42:48 AM UTC. 2024-01-13T11:44:27.120 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:44:27.312 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:44:27.312 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-01-13T11:44:27.312 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:44:27.312 INFO:teuthology.orchestra.run.smithi123.stdout:Installing group/module packages: 2024-01-13T11:44:27.313 INFO:teuthology.orchestra.run.smithi123.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-13T11:44:27.313 INFO:teuthology.orchestra.run.smithi123.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-13T11:44:27.313 INFO:teuthology.orchestra.run.smithi123.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-13T11:44:27.313 INFO:teuthology.orchestra.run.smithi123.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-13T11:44:27.313 INFO:teuthology.orchestra.run.smithi123.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-13T11:44:27.313 INFO:teuthology.orchestra.run.smithi123.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-13T11:44:27.313 INFO:teuthology.orchestra.run.smithi123.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-13T11:44:27.313 INFO:teuthology.orchestra.run.smithi123.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-13T11:44:27.313 INFO:teuthology.orchestra.run.smithi123.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-13T11:44:27.313 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-01-13T11:44:27.313 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-13T11:44:27.313 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-13T11:44:27.313 INFO:teuthology.orchestra.run.smithi123.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-13T11:44:27.313 INFO:teuthology.orchestra.run.smithi123.stdout:Installing weak dependencies: 2024-01-13T11:44:27.314 INFO:teuthology.orchestra.run.smithi123.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-13T11:44:27.314 INFO:teuthology.orchestra.run.smithi123.stdout:Installing module profiles: 2024-01-13T11:44:27.314 INFO:teuthology.orchestra.run.smithi123.stdout: container-tools/common 2024-01-13T11:44:27.314 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:44:27.314 INFO:teuthology.orchestra.run.smithi123.stdout:Enabling module streams: 2024-01-13T11:44:27.314 INFO:teuthology.orchestra.run.smithi123.stdout: container-tools rhel8 2024-01-13T11:44:27.314 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:44:27.314 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-13T11:44:27.314 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:44:27.314 INFO:teuthology.orchestra.run.smithi123.stdout:Install 13 Packages 2024-01-13T11:44:27.314 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:44:27.315 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 27 M 2024-01-13T11:44:27.315 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 89 M 2024-01-13T11:44:27.315 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-01-13T11:44:27.489 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:44:27.697 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:44:27.697 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-01-13T11:44:27.697 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:44:27.697 INFO:teuthology.orchestra.run.smithi059.stdout:Installing group/module packages: 2024-01-13T11:44:27.697 INFO:teuthology.orchestra.run.smithi059.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-13T11:44:27.697 INFO:teuthology.orchestra.run.smithi059.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-13T11:44:27.697 INFO:teuthology.orchestra.run.smithi059.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-13T11:44:27.697 INFO:teuthology.orchestra.run.smithi059.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-13T11:44:27.698 INFO:teuthology.orchestra.run.smithi059.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-13T11:44:27.698 INFO:teuthology.orchestra.run.smithi059.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-13T11:44:27.698 INFO:teuthology.orchestra.run.smithi059.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-13T11:44:27.698 INFO:teuthology.orchestra.run.smithi059.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-13T11:44:27.698 INFO:teuthology.orchestra.run.smithi059.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-13T11:44:27.698 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-01-13T11:44:27.698 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-13T11:44:27.698 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-13T11:44:27.698 INFO:teuthology.orchestra.run.smithi059.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-13T11:44:27.698 INFO:teuthology.orchestra.run.smithi059.stdout:Installing weak dependencies: 2024-01-13T11:44:27.698 INFO:teuthology.orchestra.run.smithi059.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-13T11:44:27.698 INFO:teuthology.orchestra.run.smithi059.stdout:Installing module profiles: 2024-01-13T11:44:27.698 INFO:teuthology.orchestra.run.smithi059.stdout: container-tools/common 2024-01-13T11:44:27.698 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:44:27.699 INFO:teuthology.orchestra.run.smithi059.stdout:Enabling module streams: 2024-01-13T11:44:27.699 INFO:teuthology.orchestra.run.smithi059.stdout: container-tools rhel8 2024-01-13T11:44:27.699 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:44:27.699 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-13T11:44:27.699 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:44:27.699 INFO:teuthology.orchestra.run.smithi059.stdout:Install 13 Packages 2024-01-13T11:44:27.699 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:44:27.699 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 27 M 2024-01-13T11:44:27.700 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 89 M 2024-01-13T11:44:27.700 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-13T11:44:27.794 INFO:teuthology.orchestra.run.smithi123.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 1.7 MB/s | 750 kB 00:00 2024-01-13T11:44:27.828 INFO:teuthology.orchestra.run.smithi123.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.1 MB/s | 1.0 MB 00:00 2024-01-13T11:44:27.861 INFO:teuthology.orchestra.run.smithi123.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 569 kB/s | 38 kB 00:00 2024-01-13T11:44:27.911 INFO:teuthology.orchestra.run.smithi123.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.8 MB/s | 241 kB 00:00 2024-01-13T11:44:27.995 INFO:teuthology.orchestra.run.smithi123.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 2.1 MB/s | 178 kB 00:00 2024-01-13T11:44:28.062 INFO:teuthology.orchestra.run.smithi123.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 371 kB/s | 25 kB 00:00 2024-01-13T11:44:28.129 INFO:teuthology.orchestra.run.smithi123.stdout:(7/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 15 MB/s | 3.9 MB 00:00 2024-01-13T11:44:28.246 INFO:teuthology.orchestra.run.smithi123.stdout:(8/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 11 MB/s | 9.6 MB 00:00 2024-01-13T11:44:28.271 INFO:teuthology.orchestra.run.smithi123.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 449 kB/s | 94 kB 00:00 2024-01-13T11:44:28.382 INFO:teuthology.orchestra.run.smithi059.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 1.5 MB/s | 750 kB 00:00 2024-01-13T11:44:28.397 INFO:teuthology.orchestra.run.smithi123.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 17 MB/s | 2.5 MB 00:00 2024-01-13T11:44:28.422 INFO:teuthology.orchestra.run.smithi123.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 350 kB/s | 53 kB 00:00 2024-01-13T11:44:28.441 INFO:teuthology.orchestra.run.smithi059.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 1.8 MB/s | 1.0 MB 00:00 2024-01-13T11:44:28.475 INFO:teuthology.orchestra.run.smithi059.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 412 kB/s | 38 kB 00:00 2024-01-13T11:44:28.481 INFO:teuthology.orchestra.run.smithi123.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 489 kB/s | 41 kB 00:00 2024-01-13T11:44:28.542 INFO:teuthology.orchestra.run.smithi059.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.3 MB/s | 241 kB 00:00 2024-01-13T11:44:28.625 INFO:teuthology.orchestra.run.smithi059.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 2.1 MB/s | 178 kB 00:00 2024-01-13T11:44:28.630 INFO:teuthology.orchestra.run.smithi123.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 16 MB/s | 8.1 MB 00:00 2024-01-13T11:44:28.631 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:44:28.631 INFO:teuthology.orchestra.run.smithi123.stdout:Total 20 MB/s | 27 MB 00:01 2024-01-13T11:44:28.709 INFO:teuthology.orchestra.run.smithi059.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 298 kB/s | 25 kB 00:00 2024-01-13T11:44:28.768 INFO:teuthology.orchestra.run.smithi059.stdout:(7/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 13 MB/s | 3.9 MB 00:00 2024-01-13T11:44:28.793 INFO:teuthology.orchestra.run.smithi059.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.1 MB/s | 94 kB 00:00 2024-01-13T11:44:28.895 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-01-13T11:44:28.917 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-01-13T11:44:28.917 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-01-13T11:44:28.918 INFO:teuthology.orchestra.run.smithi059.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 9.3 MB/s | 9.6 MB 00:01 2024-01-13T11:44:28.993 INFO:teuthology.orchestra.run.smithi059.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 704 kB/s | 53 kB 00:00 2024-01-13T11:44:29.052 INFO:teuthology.orchestra.run.smithi059.stdout:(11/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 9.8 MB/s | 2.5 MB 00:00 2024-01-13T11:44:29.077 INFO:teuthology.orchestra.run.smithi059.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 490 kB/s | 41 kB 00:00 2024-01-13T11:44:29.163 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-01-13T11:44:29.167 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-01-13T11:44:29.203 INFO:teuthology.orchestra.run.smithi059.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 19 MB/s | 8.1 MB 00:00 2024-01-13T11:44:29.203 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:44:29.203 INFO:teuthology.orchestra.run.smithi059.stdout:Total 18 MB/s | 27 MB 00:01 2024-01-13T11:44:29.483 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-13T11:44:29.507 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-13T11:44:29.507 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-13T11:44:29.779 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-13T11:44:29.782 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-13T11:44:29.842 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-01-13T11:44:30.075 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-13T11:44:30.311 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-13T11:44:30.498 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-13T11:44:30.544 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-13T11:44:30.736 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-13T11:44:30.772 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-13T11:44:30.958 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-13T11:44:31.170 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-13T11:44:31.281 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-13T11:44:31.402 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-13T11:44:32.012 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-13T11:44:32.262 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-13T11:44:32.467 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-13T11:44:32.664 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-13T11:44:32.924 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-13T11:44:33.077 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-13T11:44:33.302 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-13T11:44:33.307 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-13T11:44:33.528 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-13T11:44:33.788 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-13T11:44:34.211 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-13T11:44:34.231 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-13T11:44:34.436 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-13T11:44:34.660 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-13T11:44:35.090 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-13T11:44:35.090 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-13T11:44:35.090 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-13T11:44:35.090 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-13T11:44:35.090 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-13T11:44:35.090 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-13T11:44:35.090 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-13T11:44:35.091 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-13T11:44:35.091 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-13T11:44:35.091 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-13T11:44:35.091 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-13T11:44:35.091 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-13T11:44:35.091 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-13T11:44:35.173 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-13T11:44:35.417 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-13T11:44:35.632 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-13T11:44:36.063 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-13T11:44:36.063 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-13T11:44:36.063 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-13T11:44:36.063 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-13T11:44:36.063 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-13T11:44:36.063 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-13T11:44:36.063 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-13T11:44:36.063 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-13T11:44:36.063 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-13T11:44:36.063 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-13T11:44:36.063 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-13T11:44:36.064 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-13T11:44:36.064 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-13T11:44:36.245 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-13T11:44:36.245 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:44:36.246 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-01-13T11:44:36.246 INFO:teuthology.orchestra.run.smithi123.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-13T11:44:36.246 INFO:teuthology.orchestra.run.smithi123.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-13T11:44:36.246 INFO:teuthology.orchestra.run.smithi123.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-13T11:44:36.246 INFO:teuthology.orchestra.run.smithi123.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-13T11:44:36.246 INFO:teuthology.orchestra.run.smithi123.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-13T11:44:36.246 INFO:teuthology.orchestra.run.smithi123.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-13T11:44:36.246 INFO:teuthology.orchestra.run.smithi123.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-13T11:44:36.246 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-13T11:44:36.246 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-13T11:44:36.246 INFO:teuthology.orchestra.run.smithi123.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-13T11:44:36.246 INFO:teuthology.orchestra.run.smithi123.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-13T11:44:36.246 INFO:teuthology.orchestra.run.smithi123.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-13T11:44:36.246 INFO:teuthology.orchestra.run.smithi123.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-13T11:44:36.246 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:44:36.247 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:44:36.519 DEBUG:teuthology.parallel:result is None 2024-01-13T11:44:37.000 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-13T11:44:37.000 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:44:37.000 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-13T11:44:37.000 INFO:teuthology.orchestra.run.smithi059.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-13T11:44:37.000 INFO:teuthology.orchestra.run.smithi059.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-13T11:44:37.000 INFO:teuthology.orchestra.run.smithi059.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-13T11:44:37.000 INFO:teuthology.orchestra.run.smithi059.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-13T11:44:37.000 INFO:teuthology.orchestra.run.smithi059.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-13T11:44:37.000 INFO:teuthology.orchestra.run.smithi059.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-13T11:44:37.000 INFO:teuthology.orchestra.run.smithi059.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-13T11:44:37.000 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-13T11:44:37.001 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-13T11:44:37.001 INFO:teuthology.orchestra.run.smithi059.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-13T11:44:37.001 INFO:teuthology.orchestra.run.smithi059.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-13T11:44:37.001 INFO:teuthology.orchestra.run.smithi059.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-13T11:44:37.001 INFO:teuthology.orchestra.run.smithi059.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-13T11:44:37.001 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:44:37.001 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:44:37.257 DEBUG:teuthology.parallel:result is None 2024-01-13T11:44:37.257 INFO:teuthology.run_tasks:Running task pexec... 2024-01-13T11:44:37.266 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-13T11:44:37.266 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi059.front.sepia.ceph.com 2024-01-13T11:44:37.267 DEBUG:teuthology.orchestra.run.smithi059:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-13T11:44:37.267 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi123.front.sepia.ceph.com 2024-01-13T11:44:37.267 DEBUG:teuthology.orchestra.run.smithi123:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-13T11:44:37.422 DEBUG:teuthology.parallel:result is None 2024-01-13T11:44:37.674 DEBUG:teuthology.parallel:result is None 2024-01-13T11:44:37.674 INFO:teuthology.run_tasks:Running task install... 2024-01-13T11:44:37.683 DEBUG:teuthology.task.install:project ceph 2024-01-13T11:44:37.683 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '445562ab4bc3ddfb386936119050695810860bcb'}} 2024-01-13T11:44:37.683 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '445562ab4bc3ddfb386936119050695810860bcb'} 2024-01-13T11:44:37.683 INFO:teuthology.task.install:Using flavor: default 2024-01-13T11:44:37.691 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-13T11:44:37.691 INFO:teuthology.task.install:extra packages: [] 2024-01-13T11:44:37.691 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '445562ab4bc3ddfb386936119050695810860bcb', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-13T11:44:37.691 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-13T11:44:37.691 INFO:teuthology.packaging:ref: None 2024-01-13T11:44:37.692 INFO:teuthology.packaging:tag: v16.2.4 2024-01-13T11:44:37.692 INFO:teuthology.packaging:branch: None 2024-01-13T11:44:37.692 INFO:teuthology.packaging:sha1: 445562ab4bc3ddfb386936119050695810860bcb 2024-01-13T11:44:38.038 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-13T11:44:38.039 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-13T11:44:38.041 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '445562ab4bc3ddfb386936119050695810860bcb', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-13T11:44:38.042 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-13T11:44:38.042 INFO:teuthology.packaging:ref: None 2024-01-13T11:44:38.042 INFO:teuthology.packaging:tag: v16.2.4 2024-01-13T11:44:38.042 INFO:teuthology.packaging:branch: None 2024-01-13T11:44:38.042 INFO:teuthology.packaging:sha1: 445562ab4bc3ddfb386936119050695810860bcb 2024-01-13T11:44:38.042 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-13T11:44:38.195 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-13T11:44:38.195 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-13T11:44:38.234 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-13T11:44:38.235 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-13T11:44:38.235 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-13T11:44:38.270 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-13T11:44:38.270 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-13T11:44:38.270 INFO:teuthology.packaging:ref: None 2024-01-13T11:44:38.270 INFO:teuthology.packaging:tag: v16.2.4 2024-01-13T11:44:38.270 INFO:teuthology.packaging:branch: None 2024-01-13T11:44:38.270 INFO:teuthology.packaging:sha1: 445562ab4bc3ddfb386936119050695810860bcb 2024-01-13T11:44:38.270 DEBUG:teuthology.orchestra.run.smithi059:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-13T11:44:38.341 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-13T11:44:38.341 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-13T11:44:38.345 DEBUG:teuthology.orchestra.run.smithi059:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-13T11:44:38.367 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-13T11:44:38.367 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-01-13T11:44:38.367 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-13T11:44:38.401 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-13T11:44:38.401 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-13T11:44:38.402 INFO:teuthology.packaging:ref: None 2024-01-13T11:44:38.402 INFO:teuthology.packaging:tag: v16.2.4 2024-01-13T11:44:38.402 INFO:teuthology.packaging:branch: None 2024-01-13T11:44:38.402 INFO:teuthology.packaging:sha1: 445562ab4bc3ddfb386936119050695810860bcb 2024-01-13T11:44:38.402 DEBUG:teuthology.orchestra.run.smithi123:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-13T11:44:38.428 DEBUG:teuthology.orchestra.run.smithi059:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-13T11:44:38.475 DEBUG:teuthology.orchestra.run.smithi123:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-13T11:44:38.499 INFO:teuthology.orchestra.run.smithi059.stdout:check_obsoletes = 1 2024-01-13T11:44:38.500 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum clean all 2024-01-13T11:44:38.560 DEBUG:teuthology.orchestra.run.smithi123:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-13T11:44:38.630 INFO:teuthology.orchestra.run.smithi123.stdout:check_obsoletes = 1 2024-01-13T11:44:38.631 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum clean all 2024-01-13T11:44:38.786 INFO:teuthology.orchestra.run.smithi059.stdout:76 files removed 2024-01-13T11:44:38.815 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-radosgw 2024-01-13T11:44:38.901 INFO:teuthology.orchestra.run.smithi123.stdout:76 files removed 2024-01-13T11:44:38.929 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-radosgw 2024-01-13T11:44:39.328 INFO:teuthology.orchestra.run.smithi059.stdout:ceph packages for x86_64 866 kB/s | 74 kB 00:00 2024-01-13T11:44:39.414 INFO:teuthology.orchestra.run.smithi123.stdout:ceph packages for x86_64 979 kB/s | 74 kB 00:00 2024-01-13T11:44:39.504 INFO:teuthology.orchestra.run.smithi059.stdout:ceph noarch packages 113 kB/s | 15 kB 00:00 2024-01-13T11:44:39.531 INFO:teuthology.orchestra.run.smithi123.stdout:ceph noarch packages 192 kB/s | 15 kB 00:00 2024-01-13T11:44:39.613 INFO:teuthology.orchestra.run.smithi059.stdout:ceph source packages 22 kB/s | 1.7 kB 00:00 2024-01-13T11:44:39.648 INFO:teuthology.orchestra.run.smithi123.stdout:ceph source packages 20 kB/s | 1.7 kB 00:00 2024-01-13T11:44:41.563 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - AppStream 18 MB/s | 35 MB 00:01 2024-01-13T11:44:41.899 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - AppStream 16 MB/s | 35 MB 00:02 2024-01-13T11:44:50.831 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 58 MB 00:02 2024-01-13T11:44:50.882 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 58 MB 00:02 2024-01-13T11:44:59.232 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras 124 kB/s | 18 kB 00:00 2024-01-13T11:44:59.758 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 6.9 kB 00:00 2024-01-13T11:45:00.022 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - Extras 61 kB/s | 18 kB 00:00 2024-01-13T11:45:00.598 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2024-01-13T11:45:01.592 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - PowerTools 4.0 MB/s | 6.7 MB 00:01 2024-01-13T11:45:01.799 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - PowerTools 6.4 MB/s | 6.7 MB 00:01 2024-01-13T11:45:03.219 INFO:teuthology.orchestra.run.smithi123.stdout:Copr repo for python3-asyncssh owned by ceph 11 kB/s | 3.5 kB 00:00 2024-01-13T11:45:03.326 INFO:teuthology.orchestra.run.smithi059.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-01-13T11:45:05.918 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 6.7 MB/s | 16 MB 00:02 2024-01-13T11:45:09.968 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 439 kB/s | 24 kB 00:00 2024-01-13T11:45:11.297 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 2.0 MB/s | 16 MB 00:07 2024-01-13T11:45:14.878 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:45:14.881 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:45:14.881 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-01-13T11:45:14.881 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:45:14.881 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-13T11:45:14.881 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-13T11:45:14.881 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrading: 2024-01-13T11:45:14.881 INFO:teuthology.orchestra.run.smithi059.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-13T11:45:14.882 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-13T11:45:14.882 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-01-13T11:45:14.882 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-13T11:45:14.882 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-13T11:45:14.882 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-13T11:45:14.882 INFO:teuthology.orchestra.run.smithi059.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-13T11:45:14.882 INFO:teuthology.orchestra.run.smithi059.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-13T11:45:14.882 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-13T11:45:14.882 INFO:teuthology.orchestra.run.smithi059.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-13T11:45:14.882 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-13T11:45:14.882 INFO:teuthology.orchestra.run.smithi059.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-13T11:45:14.882 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-13T11:45:14.883 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2024-01-13T11:45:14.883 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-13T11:45:14.883 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-13T11:45:14.883 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-13T11:45:14.883 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-13T11:45:14.883 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-13T11:45:14.883 INFO:teuthology.orchestra.run.smithi059.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-13T11:45:14.883 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-13T11:45:14.883 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-13T11:45:14.883 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-13T11:45:14.883 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-13T11:45:14.884 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-13T11:45:14.884 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-13T11:45:14.884 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-13T11:45:14.884 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:45:14.884 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-13T11:45:14.884 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:45:14.884 INFO:teuthology.orchestra.run.smithi059.stdout:Install 25 Packages 2024-01-13T11:45:14.884 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrade 2 Packages 2024-01-13T11:45:14.884 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:45:14.885 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 59 M 2024-01-13T11:45:14.885 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-13T11:45:15.169 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 441 kB/s | 24 kB 00:00 2024-01-13T11:45:15.453 INFO:teuthology.orchestra.run.smithi059.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 30 MB/s | 6.0 MB 00:00 2024-01-13T11:45:15.478 INFO:teuthology.orchestra.run.smithi059.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 996 kB/s | 25 kB 00:00 2024-01-13T11:45:15.520 INFO:teuthology.orchestra.run.smithi059.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-01-13T11:45:15.554 INFO:teuthology.orchestra.run.smithi059.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-13T11:45:15.687 INFO:teuthology.orchestra.run.smithi059.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 31 MB/s | 4.1 MB 00:00 2024-01-13T11:45:15.713 INFO:teuthology.orchestra.run.smithi059.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.7 MB/s | 44 kB 00:00 2024-01-13T11:45:15.738 INFO:teuthology.orchestra.run.smithi059.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-01-13T11:45:15.764 INFO:teuthology.orchestra.run.smithi059.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2024-01-13T11:45:15.797 INFO:teuthology.orchestra.run.smithi059.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-01-13T11:45:15.831 INFO:teuthology.orchestra.run.smithi059.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2024-01-13T11:45:15.897 INFO:teuthology.orchestra.run.smithi059.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 1.7 MB/s | 114 kB 00:00 2024-01-13T11:45:16.048 INFO:teuthology.orchestra.run.smithi059.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 16 MB/s | 13 MB 00:00 2024-01-13T11:45:16.323 INFO:teuthology.orchestra.run.smithi059.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 22 MB/s | 24 MB 00:01 2024-01-13T11:45:16.380 INFO:teuthology.orchestra.run.smithi059.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 301 kB/s | 145 kB 00:00 2024-01-13T11:45:16.474 INFO:teuthology.orchestra.run.smithi059.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 472 kB/s | 44 kB 00:00 2024-01-13T11:45:16.724 INFO:teuthology.orchestra.run.smithi059.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 169 kB/s | 42 kB 00:00 2024-01-13T11:45:16.766 INFO:teuthology.orchestra.run.smithi059.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 587 kB/s | 259 kB 00:00 2024-01-13T11:45:16.791 INFO:teuthology.orchestra.run.smithi059.stdout:(18/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 701 kB/s | 47 kB 00:00 2024-01-13T11:45:16.817 INFO:teuthology.orchestra.run.smithi059.stdout:(19/27): librdkafka-1.6.1-1.el8.x86_64.rpm 881 kB/s | 677 kB 00:00 2024-01-13T11:45:16.942 INFO:teuthology.orchestra.run.smithi059.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 457 kB/s | 80 kB 00:00 2024-01-13T11:45:17.084 INFO:teuthology.orchestra.run.smithi059.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 398 kB/s | 116 kB 00:00 2024-01-13T11:45:17.142 INFO:teuthology.orchestra.run.smithi059.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 941 kB/s | 306 kB 00:00 2024-01-13T11:45:17.159 INFO:teuthology.orchestra.run.smithi059.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 840 kB/s | 59 kB 00:00 2024-01-13T11:45:17.185 INFO:teuthology.orchestra.run.smithi059.stdout:(24/27): libunwind-1.3.1-3.el8.x86_64.rpm 1.8 MB/s | 75 kB 00:00 2024-01-13T11:45:17.285 INFO:teuthology.orchestra.run.smithi059.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 30 MB/s | 3.7 MB 00:00 2024-01-13T11:45:17.310 INFO:teuthology.orchestra.run.smithi059.stdout:(26/27): leveldb-1.22-1.el8.x86_64.rpm 492 kB/s | 181 kB 00:00 2024-01-13T11:45:17.394 INFO:teuthology.orchestra.run.smithi059.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 19 MB/s | 4.0 MB 00:00 2024-01-13T11:45:17.399 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:45:17.399 INFO:teuthology.orchestra.run.smithi059.stdout:Total 23 MB/s | 59 MB 00:02 2024-01-13T11:45:17.470 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-13T11:45:17.523 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-13T11:45:17.523 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-13T11:45:17.890 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-13T11:45:17.893 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-13T11:45:19.121 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-13T11:45:19.361 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-13T11:45:19.566 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-13T11:45:19.717 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-13T11:45:19.902 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:45:19.905 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:45:19.905 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-01-13T11:45:19.905 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:45:19.905 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-01-13T11:45:19.905 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-13T11:45:19.905 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrading: 2024-01-13T11:45:19.906 INFO:teuthology.orchestra.run.smithi123.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-13T11:45:19.906 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-13T11:45:19.906 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-01-13T11:45:19.906 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-13T11:45:19.906 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-13T11:45:19.906 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-13T11:45:19.906 INFO:teuthology.orchestra.run.smithi123.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-13T11:45:19.906 INFO:teuthology.orchestra.run.smithi123.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-13T11:45:19.906 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-13T11:45:19.906 INFO:teuthology.orchestra.run.smithi123.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-13T11:45:19.906 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-13T11:45:19.906 INFO:teuthology.orchestra.run.smithi123.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-13T11:45:19.906 INFO:teuthology.orchestra.run.smithi123.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-13T11:45:19.906 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2024-01-13T11:45:19.907 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-13T11:45:19.907 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-13T11:45:19.907 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-13T11:45:19.907 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-13T11:45:19.907 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-13T11:45:19.907 INFO:teuthology.orchestra.run.smithi123.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-13T11:45:19.907 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-13T11:45:19.907 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-13T11:45:19.907 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-13T11:45:19.907 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-13T11:45:19.907 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-13T11:45:19.907 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-13T11:45:19.907 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-13T11:45:19.908 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:45:19.908 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-13T11:45:19.908 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:45:19.908 INFO:teuthology.orchestra.run.smithi123.stdout:Install 25 Packages 2024-01-13T11:45:19.908 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrade 2 Packages 2024-01-13T11:45:19.908 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:45:19.908 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 59 M 2024-01-13T11:45:19.908 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-01-13T11:45:20.182 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-13T11:45:20.370 INFO:teuthology.orchestra.run.smithi123.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 43 MB/s | 6.0 MB 00:00 2024-01-13T11:45:20.387 INFO:teuthology.orchestra.run.smithi123.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 25 kB 00:00 2024-01-13T11:45:20.429 INFO:teuthology.orchestra.run.smithi123.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-01-13T11:45:20.434 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-13T11:45:20.463 INFO:teuthology.orchestra.run.smithi123.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-13T11:45:20.531 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-13T11:45:20.563 INFO:teuthology.orchestra.run.smithi123.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2024-01-13T11:45:20.580 INFO:teuthology.orchestra.run.smithi123.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 2.6 MB/s | 44 kB 00:00 2024-01-13T11:45:20.606 INFO:teuthology.orchestra.run.smithi123.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-01-13T11:45:20.631 INFO:teuthology.orchestra.run.smithi123.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2024-01-13T11:45:20.665 INFO:teuthology.orchestra.run.smithi123.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-01-13T11:45:20.698 INFO:teuthology.orchestra.run.smithi123.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2024-01-13T11:45:20.724 INFO:teuthology.orchestra.run.smithi123.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-01-13T11:45:20.788 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-13T11:45:20.874 INFO:teuthology.orchestra.run.smithi123.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 20 MB/s | 13 MB 00:00 2024-01-13T11:45:20.959 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-13T11:45:21.141 INFO:teuthology.orchestra.run.smithi123.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 26 MB/s | 24 MB 00:00 2024-01-13T11:45:21.157 INFO:teuthology.orchestra.run.smithi123.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 335 kB/s | 145 kB 00:00 2024-01-13T11:45:21.173 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-13T11:45:21.241 INFO:teuthology.orchestra.run.smithi123.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 527 kB/s | 44 kB 00:00 2024-01-13T11:45:21.419 INFO:teuthology.orchestra.run.smithi123.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 238 kB/s | 42 kB 00:00 2024-01-13T11:45:21.423 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2024-01-13T11:45:21.475 INFO:teuthology.orchestra.run.smithi123.stdout:(17/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 838 kB/s | 47 kB 00:00 2024-01-13T11:45:21.500 INFO:teuthology.orchestra.run.smithi123.stdout:(18/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 723 kB/s | 259 kB 00:00 2024-01-13T11:45:21.526 INFO:teuthology.orchestra.run.smithi123.stdout:(19/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.0 MB/s | 677 kB 00:00 2024-01-13T11:45:21.551 INFO:teuthology.orchestra.run.smithi123.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 1.0 MB/s | 80 kB 00:00 2024-01-13T11:45:22.011 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-13T11:45:22.063 INFO:teuthology.orchestra.run.smithi123.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 207 kB/s | 116 kB 00:00 2024-01-13T11:45:22.110 INFO:teuthology.orchestra.run.smithi123.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 324 kB/s | 181 kB 00:00 2024-01-13T11:45:22.150 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-13T11:45:22.202 INFO:teuthology.orchestra.run.smithi123.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 453 kB/s | 306 kB 00:00 2024-01-13T11:45:22.227 INFO:teuthology.orchestra.run.smithi123.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 360 kB/s | 59 kB 00:00 2024-01-13T11:45:22.242 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-13T11:45:22.302 INFO:teuthology.orchestra.run.smithi123.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 37 MB/s | 3.7 MB 00:00 2024-01-13T11:45:22.327 INFO:teuthology.orchestra.run.smithi123.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 345 kB/s | 75 kB 00:00 2024-01-13T11:45:22.394 INFO:teuthology.orchestra.run.smithi123.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 24 MB/s | 4.0 MB 00:00 2024-01-13T11:45:22.396 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:45:22.397 INFO:teuthology.orchestra.run.smithi123.stdout:Total 24 MB/s | 59 MB 00:02 2024-01-13T11:45:22.423 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-13T11:45:22.468 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-01-13T11:45:22.516 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-01-13T11:45:22.516 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-01-13T11:45:22.629 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-13T11:45:22.696 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-13T11:45:22.862 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-01-13T11:45:22.865 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-01-13T11:45:22.891 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-13T11:45:23.077 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-13T11:45:23.267 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-13T11:45:23.348 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-13T11:45:23.517 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-13T11:45:23.977 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-13T11:45:24.004 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-01-13T11:45:24.150 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-13T11:45:24.207 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-13T11:45:24.289 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-13T11:45:24.426 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-13T11:45:24.504 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-13T11:45:24.594 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-13T11:45:24.734 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-13T11:45:24.892 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-13T11:45:24.990 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-13T11:45:25.024 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-13T11:45:25.195 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-13T11:45:25.227 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-13T11:45:25.353 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-13T11:45:25.384 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-13T11:45:25.610 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-13T11:45:25.769 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-13T11:45:25.805 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-13T11:45:25.832 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-13T11:45:26.026 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-13T11:45:26.069 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2024-01-13T11:45:26.612 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-13T11:45:26.752 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-13T11:45:26.830 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-13T11:45:27.015 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-13T11:45:27.263 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-13T11:45:27.329 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-13T11:45:27.499 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-13T11:45:27.678 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-13T11:45:27.877 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-13T11:45:27.956 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-13T11:45:28.093 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-13T11:45:28.529 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-13T11:45:28.774 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-13T11:45:29.002 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-13T11:45:29.042 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-13T11:45:29.101 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-13T11:45:29.280 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-13T11:45:29.487 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-13T11:45:29.610 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-13T11:45:29.663 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-13T11:45:29.809 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-13T11:45:29.887 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-13T11:45:30.068 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-13T11:45:30.149 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-13T11:45:30.243 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-13T11:45:30.244 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-13T11:45:30.416 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-13T11:45:30.468 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-13T11:45:32.952 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-13T11:45:33.176 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-13T11:45:33.873 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-13T11:45:34.329 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-13T11:45:34.419 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-13T11:45:34.584 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-13T11:45:50.638 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-13T11:45:50.638 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /sys 2024-01-13T11:45:50.638 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /proc 2024-01-13T11:45:50.638 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /dev 2024-01-13T11:45:50.638 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /run 2024-01-13T11:45:50.638 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /mnt 2024-01-13T11:45:50.638 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /var/tmp 2024-01-13T11:45:50.638 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /home 2024-01-13T11:45:50.638 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /tmp 2024-01-13T11:45:50.639 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /dev 2024-01-13T11:45:50.639 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:45:52.071 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-13T11:45:52.180 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-13T11:45:52.180 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-13T11:45:52.407 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-13T11:45:52.407 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-13T11:45:53.145 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-13T11:45:53.145 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-13T11:45:53.145 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-13T11:45:53.145 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-13T11:45:53.145 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-13T11:45:53.145 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-13T11:45:53.145 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-13T11:45:53.146 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-13T11:45:53.146 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-13T11:45:53.146 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-13T11:45:53.146 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-13T11:45:53.146 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-13T11:45:53.146 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-13T11:45:53.146 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-13T11:45:53.146 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-13T11:45:53.146 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-13T11:45:53.146 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-13T11:45:53.146 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-13T11:45:53.146 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-13T11:45:53.146 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2024-01-13T11:45:53.146 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-13T11:45:53.147 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-13T11:45:53.147 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-13T11:45:53.147 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-13T11:45:53.147 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-13T11:45:53.147 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-13T11:45:53.147 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-13T11:45:53.147 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-13T11:45:53.147 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-13T11:45:54.234 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-13T11:45:54.234 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /sys 2024-01-13T11:45:54.234 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /proc 2024-01-13T11:45:54.234 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /dev 2024-01-13T11:45:54.234 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /run 2024-01-13T11:45:54.234 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /mnt 2024-01-13T11:45:54.234 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /var/tmp 2024-01-13T11:45:54.234 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /home 2024-01-13T11:45:54.235 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /tmp 2024-01-13T11:45:54.235 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /dev 2024-01-13T11:45:54.235 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:45:54.461 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-13T11:45:54.462 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:45:54.462 INFO:teuthology.orchestra.run.smithi059.stdout:Upgraded: 2024-01-13T11:45:54.462 INFO:teuthology.orchestra.run.smithi059.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:54.462 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-13T11:45:54.462 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:54.462 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:54.462 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:54.462 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:54.462 INFO:teuthology.orchestra.run.smithi059.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-13T11:45:54.462 INFO:teuthology.orchestra.run.smithi059.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-13T11:45:54.462 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-13T11:45:54.463 INFO:teuthology.orchestra.run.smithi059.stdout: leveldb-1.22-1.el8.x86_64 2024-01-13T11:45:54.463 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:54.463 INFO:teuthology.orchestra.run.smithi059.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-13T11:45:54.463 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-13T11:45:54.463 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-13T11:45:54.463 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:54.463 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-13T11:45:54.463 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:54.463 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-13T11:45:54.463 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-13T11:45:54.463 INFO:teuthology.orchestra.run.smithi059.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-13T11:45:54.463 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:54.463 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:54.464 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:54.464 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-13T11:45:54.464 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:54.464 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:54.464 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:54.464 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:45:54.464 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:45:55.067 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-test 2024-01-13T11:45:55.559 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-13T11:45:55.646 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-13T11:45:55.646 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-13T11:45:55.883 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-13T11:45:55.883 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-13T11:45:55.955 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:46 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:45:56.588 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-13T11:45:56.589 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-13T11:45:56.589 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-13T11:45:56.589 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-13T11:45:56.589 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-13T11:45:56.589 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-13T11:45:56.589 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-13T11:45:56.589 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-13T11:45:56.589 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-13T11:45:56.589 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-13T11:45:56.589 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-13T11:45:56.589 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-13T11:45:56.589 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-13T11:45:56.590 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-13T11:45:56.590 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-13T11:45:56.590 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-13T11:45:56.590 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-13T11:45:56.590 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-13T11:45:56.590 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-13T11:45:56.590 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2024-01-13T11:45:56.591 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-13T11:45:56.591 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-13T11:45:56.591 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-13T11:45:56.591 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-13T11:45:56.591 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-13T11:45:56.591 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-13T11:45:56.591 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-13T11:45:56.591 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-13T11:45:56.591 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-13T11:45:57.777 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:45:57.778 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:45:57.778 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-01-13T11:45:57.778 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:45:57.778 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-13T11:45:57.778 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-13T11:45:57.778 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-01-13T11:45:57.778 INFO:teuthology.orchestra.run.smithi059.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-13T11:45:57.778 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-13T11:45:57.779 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2024-01-13T11:45:57.779 INFO:teuthology.orchestra.run.smithi059.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-13T11:45:57.779 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-13T11:45:57.779 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:45:57.779 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-13T11:45:57.779 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:45:57.779 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2024-01-13T11:45:57.779 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:45:57.780 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 53 M 2024-01-13T11:45:57.780 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 229 M 2024-01-13T11:45:57.780 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-13T11:45:57.903 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-13T11:45:57.903 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:45:57.904 INFO:teuthology.orchestra.run.smithi123.stdout:Upgraded: 2024-01-13T11:45:57.904 INFO:teuthology.orchestra.run.smithi123.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:57.904 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-01-13T11:45:57.904 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:57.904 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:57.904 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:57.904 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:57.905 INFO:teuthology.orchestra.run.smithi123.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-13T11:45:57.905 INFO:teuthology.orchestra.run.smithi123.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-13T11:45:57.905 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-13T11:45:57.905 INFO:teuthology.orchestra.run.smithi123.stdout: leveldb-1.22-1.el8.x86_64 2024-01-13T11:45:57.905 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:57.905 INFO:teuthology.orchestra.run.smithi123.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-13T11:45:57.905 INFO:teuthology.orchestra.run.smithi123.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-13T11:45:57.905 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-13T11:45:57.905 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:57.905 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-13T11:45:57.905 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:57.905 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-13T11:45:57.906 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-13T11:45:57.906 INFO:teuthology.orchestra.run.smithi123.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-13T11:45:57.906 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:57.906 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:57.906 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:57.906 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-13T11:45:57.906 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:57.906 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:57.906 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-13T11:45:57.906 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:45:57.906 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:45:57.936 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 169 kB 00:00 2024-01-13T11:45:58.075 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-test 2024-01-13T11:45:58.687 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 63 MB/s | 52 MB 00:00 2024-01-13T11:45:58.712 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 240 kB/s | 203 kB 00:00 2024-01-13T11:45:58.737 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 234 kB/s | 187 kB 00:00 2024-01-13T11:45:58.829 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 605 kB/s | 71 kB 00:00 2024-01-13T11:45:58.925 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:00:43 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:45:59.161 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 682 kB/s | 323 kB 00:00 2024-01-13T11:45:59.162 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:45:59.163 INFO:teuthology.orchestra.run.smithi059.stdout:Total 38 MB/s | 53 MB 00:01 2024-01-13T11:45:59.207 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-13T11:45:59.230 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-13T11:45:59.230 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-13T11:45:59.545 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-13T11:45:59.549 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-13T11:46:00.082 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-13T11:46:00.321 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-13T11:46:00.542 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-13T11:46:00.628 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:46:00.629 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:46:00.629 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-01-13T11:46:00.629 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:46:00.629 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-01-13T11:46:00.629 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-13T11:46:00.629 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-01-13T11:46:00.629 INFO:teuthology.orchestra.run.smithi123.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-13T11:46:00.629 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-13T11:46:00.629 INFO:teuthology.orchestra.run.smithi123.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2024-01-13T11:46:00.630 INFO:teuthology.orchestra.run.smithi123.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-13T11:46:00.630 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-13T11:46:00.630 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:00.630 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-13T11:46:00.630 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:46:00.630 INFO:teuthology.orchestra.run.smithi123.stdout:Install 6 Packages 2024-01-13T11:46:00.630 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:00.631 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 53 M 2024-01-13T11:46:00.631 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 229 M 2024-01-13T11:46:00.631 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-01-13T11:46:00.720 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2024-01-13T11:46:00.803 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2024-01-13T11:46:00.960 INFO:teuthology.orchestra.run.smithi123.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.7 MB/s | 169 kB 00:00 2024-01-13T11:46:00.987 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-13T11:46:01.162 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-13T11:46:01.611 INFO:teuthology.orchestra.run.smithi123.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 73 MB/s | 52 MB 00:00 2024-01-13T11:46:01.636 INFO:teuthology.orchestra.run.smithi123.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 277 kB/s | 187 kB 00:00 2024-01-13T11:46:01.662 INFO:teuthology.orchestra.run.smithi123.stdout:(4/6): jq-1.6-8.el8.x86_64.rpm 266 kB/s | 203 kB 00:00 2024-01-13T11:46:01.787 INFO:teuthology.orchestra.run.smithi123.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 470 kB/s | 71 kB 00:00 2024-01-13T11:46:02.212 INFO:teuthology.orchestra.run.smithi123.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 538 kB/s | 323 kB 00:00 2024-01-13T11:46:02.212 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:46:02.213 INFO:teuthology.orchestra.run.smithi123.stdout:Total 33 MB/s | 53 MB 00:01 2024-01-13T11:46:02.256 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-01-13T11:46:02.276 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-01-13T11:46:02.276 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-01-13T11:46:02.576 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-01-13T11:46:02.579 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-01-13T11:46:03.067 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-01-13T11:46:03.290 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-13T11:46:03.515 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-13T11:46:03.654 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2024-01-13T11:46:03.716 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2024-01-13T11:46:04.036 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-13T11:46:04.212 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-13T11:46:06.339 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-13T11:46:06.610 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-13T11:46:06.926 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-13T11:46:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-13T11:46:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-13T11:46:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-13T11:46:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2024-01-13T11:46:06.927 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-13T11:46:07.697 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-13T11:46:07.697 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:07.697 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-13T11:46:07.697 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-13T11:46:07.697 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2024-01-13T11:46:07.697 INFO:teuthology.orchestra.run.smithi059.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-13T11:46:07.697 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:07.697 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:46:07.927 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph 2024-01-13T11:46:08.817 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:59 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:46:09.063 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-13T11:46:09.310 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-13T11:46:09.598 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-13T11:46:09.598 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-13T11:46:09.598 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-13T11:46:09.598 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-13T11:46:09.598 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2024-01-13T11:46:09.598 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-13T11:46:10.337 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-13T11:46:10.337 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:10.337 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-01-13T11:46:10.337 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-13T11:46:10.338 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2024-01-13T11:46:10.338 INFO:teuthology.orchestra.run.smithi123.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-13T11:46:10.338 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:10.338 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:46:10.549 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph 2024-01-13T11:46:10.643 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:46:10.646 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:46:10.646 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-01-13T11:46:10.647 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:46:10.647 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-13T11:46:10.647 INFO:teuthology.orchestra.run.smithi059.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-13T11:46:10.647 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-01-13T11:46:10.647 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-13T11:46:10.647 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-13T11:46:10.647 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-13T11:46:10.648 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-13T11:46:10.648 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-13T11:46:10.648 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-13T11:46:10.648 INFO:teuthology.orchestra.run.smithi059.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-13T11:46:10.648 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-13T11:46:10.648 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-13T11:46:10.648 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-13T11:46:10.648 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-13T11:46:10.648 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-13T11:46:10.649 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-13T11:46:10.649 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-13T11:46:10.649 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-13T11:46:10.649 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-13T11:46:10.649 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-13T11:46:10.649 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-13T11:46:10.649 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-13T11:46:10.649 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-13T11:46:10.649 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-13T11:46:10.650 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-13T11:46:10.650 INFO:teuthology.orchestra.run.smithi059.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-13T11:46:10.650 INFO:teuthology.orchestra.run.smithi059.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-13T11:46:10.650 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-13T11:46:10.650 INFO:teuthology.orchestra.run.smithi059.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-13T11:46:10.650 INFO:teuthology.orchestra.run.smithi059.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-13T11:46:10.650 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-13T11:46:10.650 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-13T11:46:10.650 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-13T11:46:10.651 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-13T11:46:10.651 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:10.651 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-13T11:46:10.651 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:46:10.651 INFO:teuthology.orchestra.run.smithi059.stdout:Install 32 Packages 2024-01-13T11:46:10.651 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:10.652 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 31 M 2024-01-13T11:46:10.652 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 111 M 2024-01-13T11:46:10.652 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-13T11:46:11.281 INFO:teuthology.orchestra.run.smithi059.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 124 kB/s | 6.3 kB 00:00 2024-01-13T11:46:11.356 INFO:teuthology.orchestra.run.smithi059.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-01-13T11:46:11.404 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:00:56 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:46:11.406 INFO:teuthology.orchestra.run.smithi059.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-01-13T11:46:11.431 INFO:teuthology.orchestra.run.smithi059.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2024-01-13T11:46:11.523 INFO:teuthology.orchestra.run.smithi059.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2024-01-13T11:46:11.816 INFO:teuthology.orchestra.run.smithi059.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 40 MB/s | 18 MB 00:00 2024-01-13T11:46:11.849 INFO:teuthology.orchestra.run.smithi059.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 119 kB/s | 39 kB 00:00 2024-01-13T11:46:11.874 INFO:teuthology.orchestra.run.smithi059.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 355 kB/s | 157 kB 00:00 2024-01-13T11:46:11.992 INFO:teuthology.orchestra.run.smithi059.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 402 kB/s | 57 kB 00:00 2024-01-13T11:46:12.083 INFO:teuthology.orchestra.run.smithi059.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.1 MB/s | 457 kB 00:00 2024-01-13T11:46:12.125 INFO:teuthology.orchestra.run.smithi059.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 335 kB/s | 103 kB 00:00 2024-01-13T11:46:12.234 INFO:teuthology.orchestra.run.smithi059.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.0 MB/s | 249 kB 00:00 2024-01-13T11:46:12.318 INFO:teuthology.orchestra.run.smithi059.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 568 kB/s | 109 kB 00:00 2024-01-13T11:46:12.351 INFO:teuthology.orchestra.run.smithi059.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 2.3 MB/s | 641 kB 00:00 2024-01-13T11:46:12.668 INFO:teuthology.orchestra.run.smithi059.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 100 kB/s | 44 kB 00:00 2024-01-13T11:46:12.827 INFO:teuthology.orchestra.run.smithi059.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 368 kB/s | 185 kB 00:00 2024-01-13T11:46:12.844 INFO:teuthology.orchestra.run.smithi059.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 351 kB/s | 173 kB 00:00 2024-01-13T11:46:12.927 INFO:teuthology.orchestra.run.smithi059.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2024-01-13T11:46:12.952 INFO:teuthology.orchestra.run.smithi059.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 162 kB/s | 18 kB 00:00 2024-01-13T11:46:13.036 INFO:teuthology.orchestra.run.smithi059.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.0 MB/s | 384 kB 00:00 2024-01-13T11:46:13.061 INFO:teuthology.orchestra.run.smithi059.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 364 kB/s | 49 kB 00:00 2024-01-13T11:46:13.087 INFO:teuthology.orchestra.run.smithi059.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 438 kB/s | 59 kB 00:00 2024-01-13T11:46:13.116 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:46:13.119 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:46:13.119 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-01-13T11:46:13.119 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:46:13.119 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-01-13T11:46:13.119 INFO:teuthology.orchestra.run.smithi123.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-13T11:46:13.119 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-01-13T11:46:13.119 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-13T11:46:13.119 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-13T11:46:13.119 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-13T11:46:13.119 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-13T11:46:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-13T11:46:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-13T11:46:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-13T11:46:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-13T11:46:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-13T11:46:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-13T11:46:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-13T11:46:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-13T11:46:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-13T11:46:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-13T11:46:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-13T11:46:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-13T11:46:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-13T11:46:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-13T11:46:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-13T11:46:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-13T11:46:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-13T11:46:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-13T11:46:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-13T11:46:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-13T11:46:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-13T11:46:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-13T11:46:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-13T11:46:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-13T11:46:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-13T11:46:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-13T11:46:13.122 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-13T11:46:13.122 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:13.122 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-13T11:46:13.122 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:46:13.122 INFO:teuthology.orchestra.run.smithi123.stdout:Install 32 Packages 2024-01-13T11:46:13.122 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:13.122 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 31 M 2024-01-13T11:46:13.122 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 111 M 2024-01-13T11:46:13.123 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-01-13T11:46:13.145 INFO:teuthology.orchestra.run.smithi059.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.5 MB/s | 283 kB 00:00 2024-01-13T11:46:13.171 INFO:teuthology.orchestra.run.smithi059.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 143 kB/s | 16 kB 00:00 2024-01-13T11:46:13.196 INFO:teuthology.orchestra.run.smithi059.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 172 kB/s | 19 kB 00:00 2024-01-13T11:46:13.279 INFO:teuthology.orchestra.run.smithi059.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 179 kB/s | 24 kB 00:00 2024-01-13T11:46:13.305 INFO:teuthology.orchestra.run.smithi059.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 218 kB/s | 29 kB 00:00 2024-01-13T11:46:13.330 INFO:teuthology.orchestra.run.smithi059.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 206 kB/s | 27 kB 00:00 2024-01-13T11:46:13.388 INFO:teuthology.orchestra.run.smithi059.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.3 MB/s | 255 kB 00:00 2024-01-13T11:46:13.430 INFO:teuthology.orchestra.run.smithi059.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 855 kB/s | 85 kB 00:00 2024-01-13T11:46:13.514 INFO:teuthology.orchestra.run.smithi059.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 183 kB/s | 23 kB 00:00 2024-01-13T11:46:13.539 INFO:teuthology.orchestra.run.smithi059.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.2 MB/s | 251 kB 00:00 2024-01-13T11:46:13.543 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:46:13.544 INFO:teuthology.orchestra.run.smithi059.stdout:Total 11 MB/s | 31 MB 00:02 2024-01-13T11:46:13.604 INFO:teuthology.orchestra.run.smithi123.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 137 kB/s | 6.3 kB 00:00 2024-01-13T11:46:13.631 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-13T11:46:13.654 INFO:teuthology.orchestra.run.smithi123.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-01-13T11:46:13.673 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-13T11:46:13.673 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-13T11:46:13.746 INFO:teuthology.orchestra.run.smithi123.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 13 MB/s | 2.4 MB 00:00 2024-01-13T11:46:13.771 INFO:teuthology.orchestra.run.smithi123.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2024-01-13T11:46:13.847 INFO:teuthology.orchestra.run.smithi123.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2024-01-13T11:46:14.011 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-13T11:46:14.014 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-13T11:46:14.072 INFO:teuthology.orchestra.run.smithi123.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 44 MB/s | 18 MB 00:00 2024-01-13T11:46:14.097 INFO:teuthology.orchestra.run.smithi123.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 155 kB/s | 39 kB 00:00 2024-01-13T11:46:14.123 INFO:teuthology.orchestra.run.smithi123.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 447 kB/s | 157 kB 00:00 2024-01-13T11:46:14.223 INFO:teuthology.orchestra.run.smithi123.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 455 kB/s | 57 kB 00:00 2024-01-13T11:46:14.315 INFO:teuthology.orchestra.run.smithi123.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.3 MB/s | 457 kB 00:00 2024-01-13T11:46:14.373 INFO:teuthology.orchestra.run.smithi123.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 344 kB/s | 103 kB 00:00 2024-01-13T11:46:14.432 INFO:teuthology.orchestra.run.smithi123.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.2 MB/s | 249 kB 00:00 2024-01-13T11:46:14.616 INFO:teuthology.orchestra.run.smithi123.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 450 kB/s | 109 kB 00:00 2024-01-13T11:46:14.641 INFO:teuthology.orchestra.run.smithi123.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.9 MB/s | 641 kB 00:00 2024-01-13T11:46:14.666 INFO:teuthology.orchestra.run.smithi123.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 186 kB/s | 44 kB 00:00 2024-01-13T11:46:14.825 INFO:teuthology.orchestra.run.smithi123.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 940 kB/s | 173 kB 00:00 2024-01-13T11:46:14.900 INFO:teuthology.orchestra.run.smithi123.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 652 kB/s | 185 kB 00:00 2024-01-13T11:46:14.926 INFO:teuthology.orchestra.run.smithi123.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 110 kB/s | 11 kB 00:00 2024-01-13T11:46:15.004 INFO:teuthology.orchestra.run.smithi123.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 170 kB/s | 18 kB 00:00 2024-01-13T11:46:15.043 INFO:teuthology.orchestra.run.smithi123.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 417 kB/s | 49 kB 00:00 2024-01-13T11:46:15.068 INFO:teuthology.orchestra.run.smithi123.stdout:(21/32): python3-cherrypy-18.4.0-1.el8.noarch.r 958 kB/s | 384 kB 00:00 2024-01-13T11:46:15.093 INFO:teuthology.orchestra.run.smithi123.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 660 kB/s | 59 kB 00:00 2024-01-13T11:46:15.135 INFO:teuthology.orchestra.run.smithi123.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2024-01-13T11:46:15.160 INFO:teuthology.orchestra.run.smithi123.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.3 MB/s | 283 kB 00:00 2024-01-13T11:46:15.186 INFO:teuthology.orchestra.run.smithi123.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 203 kB/s | 19 kB 00:00 2024-01-13T11:46:15.211 INFO:teuthology.orchestra.run.smithi123.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 316 kB/s | 24 kB 00:00 2024-01-13T11:46:15.236 INFO:teuthology.orchestra.run.smithi123.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 388 kB/s | 29 kB 00:00 2024-01-13T11:46:15.261 INFO:teuthology.orchestra.run.smithi123.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 364 kB/s | 27 kB 00:00 2024-01-13T11:46:15.264 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-13T11:46:15.312 INFO:teuthology.orchestra.run.smithi123.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.3 MB/s | 251 kB 00:00 2024-01-13T11:46:15.337 INFO:teuthology.orchestra.run.smithi123.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.0 MB/s | 255 kB 00:00 2024-01-13T11:46:15.362 INFO:teuthology.orchestra.run.smithi123.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 852 kB/s | 85 kB 00:00 2024-01-13T11:46:15.387 INFO:teuthology.orchestra.run.smithi123.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 306 kB/s | 23 kB 00:00 2024-01-13T11:46:15.388 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:46:15.389 INFO:teuthology.orchestra.run.smithi123.stdout:Total 14 MB/s | 31 MB 00:02 2024-01-13T11:46:15.473 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-01-13T11:46:15.512 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-01-13T11:46:15.512 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-01-13T11:46:15.521 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-13T11:46:15.745 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-13T11:46:15.811 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-01-13T11:46:15.814 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-01-13T11:46:16.014 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-13T11:46:16.251 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-13T11:46:16.457 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-13T11:46:16.659 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-13T11:46:16.847 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-13T11:46:16.904 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-01-13T11:46:17.034 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-13T11:46:17.107 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-13T11:46:17.275 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-13T11:46:17.301 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-13T11:46:17.520 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-13T11:46:17.543 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-13T11:46:17.724 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-13T11:46:17.872 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-13T11:46:17.891 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-13T11:46:18.027 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-13T11:46:18.167 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-13T11:46:18.229 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-13T11:46:18.398 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-13T11:46:18.588 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-13T11:46:18.610 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-13T11:46:18.793 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-13T11:46:18.891 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-13T11:46:19.040 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-13T11:46:19.094 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-13T11:46:19.279 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-13T11:46:19.416 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-13T11:46:19.656 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-13T11:46:19.764 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-13T11:46:19.921 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-13T11:46:20.076 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-13T11:46:20.116 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-13T11:46:20.365 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-13T11:46:20.380 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-13T11:46:20.535 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-13T11:46:20.591 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-13T11:46:20.833 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-13T11:46:20.851 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-13T11:46:21.074 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-13T11:46:21.099 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-13T11:46:21.278 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-13T11:46:21.367 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-13T11:46:21.506 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-13T11:46:21.637 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-13T11:46:21.775 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-13T11:46:21.971 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-13T11:46:22.026 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-13T11:46:22.227 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-13T11:46:22.388 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-13T11:46:22.578 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-13T11:46:22.780 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-13T11:46:22.899 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-13T11:46:23.013 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-13T11:46:23.300 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-13T11:46:23.529 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-13T11:46:24.659 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-13T11:46:24.900 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-13T11:46:25.156 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-13T11:46:25.361 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-13T11:46:25.429 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-13T11:46:25.864 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-13T11:46:26.079 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-13T11:46:26.303 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-13T11:46:26.408 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-13T11:46:26.526 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-13T11:46:26.544 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-13T11:46:26.650 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-13T11:46:26.667 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-13T11:46:26.826 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-13T11:46:26.845 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-13T11:46:27.001 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-13T11:46:27.074 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-13T11:46:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-13T11:46:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-13T11:46:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-13T11:46:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-13T11:46:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-13T11:46:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-13T11:46:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-13T11:46:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-13T11:46:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-13T11:46:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-13T11:46:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-13T11:46:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-13T11:46:27.075 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-13T11:46:27.076 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-13T11:46:27.076 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-13T11:46:27.076 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-13T11:46:27.076 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-13T11:46:27.076 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-13T11:46:27.076 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-13T11:46:27.076 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-13T11:46:27.076 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-13T11:46:27.076 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-13T11:46:27.076 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-13T11:46:27.076 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-13T11:46:27.076 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-13T11:46:27.076 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-13T11:46:27.077 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-13T11:46:27.077 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-13T11:46:27.077 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-13T11:46:27.077 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-13T11:46:27.077 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-13T11:46:27.435 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-13T11:46:27.436 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-13T11:46:27.436 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-13T11:46:27.436 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-13T11:46:27.436 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-13T11:46:27.436 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-13T11:46:27.436 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-13T11:46:27.436 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-13T11:46:27.436 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-13T11:46:27.436 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-13T11:46:27.436 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-13T11:46:27.436 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-13T11:46:27.437 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-13T11:46:27.437 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-13T11:46:27.437 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-13T11:46:27.437 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-13T11:46:27.437 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-13T11:46:27.437 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-13T11:46:27.437 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-13T11:46:27.437 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-13T11:46:27.437 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-13T11:46:27.437 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-13T11:46:27.437 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-13T11:46:27.437 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-13T11:46:27.438 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-13T11:46:27.438 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-13T11:46:27.438 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-13T11:46:27.438 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-13T11:46:27.438 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-13T11:46:27.438 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-13T11:46:27.438 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-13T11:46:27.438 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-13T11:46:28.296 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-13T11:46:28.296 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:28.296 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-01-13T11:46:28.297 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-13T11:46:28.297 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-13T11:46:28.297 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-13T11:46:28.297 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-13T11:46:28.297 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-13T11:46:28.297 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-13T11:46:28.297 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-13T11:46:28.298 INFO:teuthology.orchestra.run.smithi123.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-13T11:46:28.298 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-13T11:46:28.298 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-13T11:46:28.298 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-13T11:46:28.298 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-13T11:46:28.298 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-13T11:46:28.298 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-13T11:46:28.298 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-13T11:46:28.298 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-13T11:46:28.298 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-13T11:46:28.299 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-13T11:46:28.299 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-13T11:46:28.299 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend-2.6-1.el8.noarch 2024-01-13T11:46:28.299 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-13T11:46:28.299 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-13T11:46:28.299 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-13T11:46:28.299 INFO:teuthology.orchestra.run.smithi123.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-13T11:46:28.299 INFO:teuthology.orchestra.run.smithi123.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-13T11:46:28.299 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-13T11:46:28.299 INFO:teuthology.orchestra.run.smithi123.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-13T11:46:28.299 INFO:teuthology.orchestra.run.smithi123.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-13T11:46:28.299 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-13T11:46:28.300 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-13T11:46:28.300 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-13T11:46:28.300 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-13T11:46:28.300 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:28.300 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:46:28.443 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-base 2024-01-13T11:46:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-13T11:46:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:28.715 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-13T11:46:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-13T11:46:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-13T11:46:28.715 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-13T11:46:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-13T11:46:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-13T11:46:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-13T11:46:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-13T11:46:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-13T11:46:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-13T11:46:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-13T11:46:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-13T11:46:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-13T11:46:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-13T11:46:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-13T11:46:28.716 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-13T11:46:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-13T11:46:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-13T11:46:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-13T11:46:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-13T11:46:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend-2.6-1.el8.noarch 2024-01-13T11:46:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-13T11:46:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-13T11:46:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-13T11:46:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-13T11:46:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-13T11:46:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-13T11:46:28.717 INFO:teuthology.orchestra.run.smithi059.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-13T11:46:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-13T11:46:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-13T11:46:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-13T11:46:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-13T11:46:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-13T11:46:28.718 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:28.718 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:46:28.913 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-base 2024-01-13T11:46:29.296 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:14 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:46:29.805 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:20 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:46:30.788 INFO:teuthology.orchestra.run.smithi123.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-13T11:46:31.026 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:46:31.027 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-01-13T11:46:31.027 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:46:31.134 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install cephadm 2024-01-13T11:46:31.380 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-13T11:46:31.636 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:46:31.637 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-01-13T11:46:31.637 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:46:31.839 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install cephadm 2024-01-13T11:46:31.984 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:16 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:46:32.729 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:23 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:46:33.693 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:46:33.694 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:46:33.694 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-01-13T11:46:33.694 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:46:33.694 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-01-13T11:46:33.694 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-13T11:46:33.694 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:33.694 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-13T11:46:33.695 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:46:33.695 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-01-13T11:46:33.695 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:33.695 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 71 k 2024-01-13T11:46:33.695 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 290 k 2024-01-13T11:46:33.695 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-01-13T11:46:33.750 INFO:teuthology.orchestra.run.smithi123.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.3 MB/s | 71 kB 00:00 2024-01-13T11:46:33.751 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:46:33.751 INFO:teuthology.orchestra.run.smithi123.stdout:Total 1.2 MB/s | 71 kB 00:00 2024-01-13T11:46:33.751 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-01-13T11:46:33.758 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-01-13T11:46:33.758 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-01-13T11:46:33.759 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-01-13T11:46:33.763 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-01-13T11:46:33.803 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-01-13T11:46:34.295 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-13T11:46:34.473 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-13T11:46:34.553 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:46:34.553 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:46:34.553 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-01-13T11:46:34.553 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:46:34.553 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-13T11:46:34.554 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-13T11:46:34.554 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:34.554 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-13T11:46:34.554 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:46:34.554 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-01-13T11:46:34.554 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:34.554 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 71 k 2024-01-13T11:46:34.554 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 290 k 2024-01-13T11:46:34.555 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-13T11:46:34.594 INFO:teuthology.orchestra.run.smithi059.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 71 kB 00:00 2024-01-13T11:46:34.594 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:46:34.595 INFO:teuthology.orchestra.run.smithi059.stdout:Total 1.7 MB/s | 71 kB 00:00 2024-01-13T11:46:34.595 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-13T11:46:34.602 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-13T11:46:34.603 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-13T11:46:34.604 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-13T11:46:34.607 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-13T11:46:34.675 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-13T11:46:34.696 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-13T11:46:35.217 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-13T11:46:35.388 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-13T11:46:35.388 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:35.388 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-01-13T11:46:35.388 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-13T11:46:35.388 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:35.388 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:46:35.411 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-13T11:46:35.644 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-13T11:46:36.364 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-13T11:46:36.365 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:36.365 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-13T11:46:36.365 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-13T11:46:36.365 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:36.365 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:46:36.385 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-immutable-object-cache 2024-01-13T11:46:36.568 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-immutable-object-cache 2024-01-13T11:46:37.249 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:22 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:46:37.504 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:28 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:46:38.964 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:46:38.964 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:46:38.964 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-01-13T11:46:38.965 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:46:38.965 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-01-13T11:46:38.965 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-13T11:46:38.965 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:38.965 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-13T11:46:38.965 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:46:38.965 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-01-13T11:46:38.965 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:38.965 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 163 k 2024-01-13T11:46:38.965 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 587 k 2024-01-13T11:46:38.965 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-01-13T11:46:39.006 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 4.0 MB/s | 163 kB 00:00 2024-01-13T11:46:39.006 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:46:39.006 INFO:teuthology.orchestra.run.smithi123.stdout:Total 3.8 MB/s | 163 kB 00:00 2024-01-13T11:46:39.007 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-01-13T11:46:39.017 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-01-13T11:46:39.017 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-01-13T11:46:39.064 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-01-13T11:46:39.067 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-01-13T11:46:39.183 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-01-13T11:46:39.319 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:46:39.320 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:46:39.320 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-01-13T11:46:39.320 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:46:39.320 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-13T11:46:39.320 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-13T11:46:39.320 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:39.320 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-13T11:46:39.320 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:46:39.320 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-01-13T11:46:39.320 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:39.321 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 163 k 2024-01-13T11:46:39.321 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 587 k 2024-01-13T11:46:39.321 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-13T11:46:39.365 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:46:39.374 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.0 MB/s | 163 kB 00:00 2024-01-13T11:46:39.375 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:46:39.375 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.9 MB/s | 163 kB 00:00 2024-01-13T11:46:39.375 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-13T11:46:39.388 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-13T11:46:39.388 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-13T11:46:39.438 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-13T11:46:39.441 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-13T11:46:39.560 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-13T11:46:39.773 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:46:39.798 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:46:40.272 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:46:40.460 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:46:40.460 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:40.461 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-01-13T11:46:40.461 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-13T11:46:40.461 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:40.461 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:46:40.688 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr 2024-01-13T11:46:40.989 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:46:40.989 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:40.989 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-13T11:46:40.989 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-13T11:46:40.989 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:40.989 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:46:41.247 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr 2024-01-13T11:46:41.546 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:26 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:46:42.145 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:33 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:46:43.039 INFO:teuthology.orchestra.run.smithi123.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-13T11:46:43.272 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:46:43.273 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-01-13T11:46:43.274 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:46:43.375 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-dashboard 2024-01-13T11:46:43.727 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-13T11:46:43.983 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:46:43.985 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-01-13T11:46:43.985 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:46:44.083 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-dashboard 2024-01-13T11:46:44.226 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:29 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:46:44.971 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:35 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:46:45.925 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:46:45.926 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:46:45.926 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-01-13T11:46:45.926 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:46:45.927 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-01-13T11:46:45.927 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-13T11:46:45.927 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-01-13T11:46:45.927 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-13T11:46:45.927 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-13T11:46:45.927 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-13T11:46:45.928 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-13T11:46:45.928 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-13T11:46:45.928 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:45.928 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-13T11:46:45.928 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:46:45.928 INFO:teuthology.orchestra.run.smithi123.stdout:Install 6 Packages 2024-01-13T11:46:45.929 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:45.929 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 3.0 M 2024-01-13T11:46:45.929 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 15 M 2024-01-13T11:46:45.929 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-01-13T11:46:46.242 INFO:teuthology.orchestra.run.smithi123.stdout:(1/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 165 kB/s | 8.9 kB 00:00 2024-01-13T11:46:46.259 INFO:teuthology.orchestra.run.smithi123.stdout:(2/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 337 kB/s | 24 kB 00:00 2024-01-13T11:46:46.317 INFO:teuthology.orchestra.run.smithi123.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 20 MB/s | 2.7 MB 00:00 2024-01-13T11:46:46.368 INFO:teuthology.orchestra.run.smithi123.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 309 kB/s | 34 kB 00:00 2024-01-13T11:46:46.426 INFO:teuthology.orchestra.run.smithi123.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 231 kB/s | 43 kB 00:00 2024-01-13T11:46:46.451 INFO:teuthology.orchestra.run.smithi123.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2024-01-13T11:46:46.453 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:46:46.453 INFO:teuthology.orchestra.run.smithi123.stdout:Total 5.6 MB/s | 3.0 MB 00:00 2024-01-13T11:46:46.463 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-01-13T11:46:46.472 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-01-13T11:46:46.473 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-01-13T11:46:46.646 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-01-13T11:46:46.650 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-01-13T11:46:46.801 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:46:46.802 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:46:46.802 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-01-13T11:46:46.802 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:46:46.802 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-13T11:46:46.803 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-13T11:46:46.803 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-01-13T11:46:46.803 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-13T11:46:46.803 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-13T11:46:46.803 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-13T11:46:46.803 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-13T11:46:46.803 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-13T11:46:46.803 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:46.803 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-13T11:46:46.803 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:46:46.803 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2024-01-13T11:46:46.803 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:46.804 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 3.0 M 2024-01-13T11:46:46.804 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 15 M 2024-01-13T11:46:46.804 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-13T11:46:46.927 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-01-13T11:46:47.144 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-13T11:46:47.368 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 438 kB/s | 24 kB 00:00 2024-01-13T11:46:47.394 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 112 kB/s | 8.9 kB 00:00 2024-01-13T11:46:47.397 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-13T11:46:47.477 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 16 MB/s | 2.7 MB 00:00 2024-01-13T11:46:47.553 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 212 kB/s | 34 kB 00:00 2024-01-13T11:46:47.586 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.9 MB/s | 196 kB 00:00 2024-01-13T11:46:47.599 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-13T11:46:47.611 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 176 kB/s | 43 kB 00:00 2024-01-13T11:46:47.615 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:46:47.615 INFO:teuthology.orchestra.run.smithi059.stdout:Total 3.6 MB/s | 3.0 MB 00:00 2024-01-13T11:46:47.628 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-13T11:46:47.639 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-13T11:46:47.639 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-13T11:46:47.744 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-13T11:46:47.806 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-13T11:46:47.810 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-13T11:46:48.121 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-13T11:46:48.365 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-13T11:46:48.676 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-13T11:46:48.804 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-13T11:46:48.958 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-13T11:46:49.068 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-13T11:46:49.274 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-13T11:46:49.357 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-13T11:46:49.357 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-13T11:46:49.357 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-13T11:46:49.357 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-13T11:46:49.357 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-13T11:46:49.357 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-13T11:46:50.063 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-13T11:46:50.064 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:50.064 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-01-13T11:46:50.064 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-13T11:46:50.064 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-13T11:46:50.064 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-13T11:46:50.064 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-13T11:46:50.064 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-13T11:46:50.064 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-13T11:46:50.064 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:50.064 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:46:50.185 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-13T11:46:50.325 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-13T11:46:50.629 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-13T11:46:50.916 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-13T11:46:50.916 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-13T11:46:50.916 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-13T11:46:50.916 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-13T11:46:50.916 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-13T11:46:50.916 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-13T11:46:51.036 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:36 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:46:51.676 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-13T11:46:51.676 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:51.676 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-13T11:46:51.676 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-13T11:46:51.676 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-13T11:46:51.676 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-13T11:46:51.676 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-13T11:46:51.676 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-13T11:46:51.676 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-13T11:46:51.676 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:51.676 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:46:51.803 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-13T11:46:52.694 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:43 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:46:52.726 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:46:52.727 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================================ 2024-01-13T11:46:52.727 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-01-13T11:46:52.728 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================================ 2024-01-13T11:46:52.728 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-01-13T11:46:52.728 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-13T11:46:52.728 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-01-13T11:46:52.728 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-13T11:46:52.728 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-13T11:46:52.728 INFO:teuthology.orchestra.run.smithi123.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-13T11:46:52.728 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-13T11:46:52.728 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-13T11:46:52.728 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-13T11:46:52.728 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-13T11:46:52.728 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:52.729 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-13T11:46:52.729 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================================ 2024-01-13T11:46:52.729 INFO:teuthology.orchestra.run.smithi123.stdout:Install 8 Packages 2024-01-13T11:46:52.729 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:46:52.729 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 37 M 2024-01-13T11:46:52.729 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 210 M 2024-01-13T11:46:52.729 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-01-13T11:46:53.148 INFO:teuthology.orchestra.run.smithi123.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 31 MB/s | 8.6 MB 00:00 2024-01-13T11:46:53.950 INFO:teuthology.orchestra.run.smithi123.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.5 MB/s | 4.9 MB 00:01 2024-01-13T11:46:54.025 INFO:teuthology.orchestra.run.smithi123.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 4.1 MB/s | 4.7 MB 00:01 2024-01-13T11:46:54.051 INFO:teuthology.orchestra.run.smithi123.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.2 MB/s | 225 kB 00:00 2024-01-13T11:46:54.117 INFO:teuthology.orchestra.run.smithi123.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.9 MB/s | 3.7 MB 00:00 2024-01-13T11:46:54.360 INFO:teuthology.orchestra.run.smithi123.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 714 kB/s | 172 kB 00:00 2024-01-13T11:46:54.451 INFO:teuthology.orchestra.run.smithi123.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.6 MB/s | 645 kB 00:00 2024-01-13T11:46:54.549 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:46:54.551 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================================ 2024-01-13T11:46:54.551 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-01-13T11:46:54.551 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================================ 2024-01-13T11:46:54.551 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-13T11:46:54.551 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-13T11:46:54.551 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-01-13T11:46:54.551 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-13T11:46:54.551 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-13T11:46:54.551 INFO:teuthology.orchestra.run.smithi059.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-13T11:46:54.551 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-13T11:46:54.552 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-13T11:46:54.552 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-13T11:46:54.552 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-13T11:46:54.552 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:54.552 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-13T11:46:54.552 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================================ 2024-01-13T11:46:54.552 INFO:teuthology.orchestra.run.smithi059.stdout:Install 8 Packages 2024-01-13T11:46:54.552 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:46:54.553 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 37 M 2024-01-13T11:46:54.553 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 210 M 2024-01-13T11:46:54.553 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-13T11:46:54.593 INFO:teuthology.orchestra.run.smithi123.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 25 MB/s | 14 MB 00:00 2024-01-13T11:46:54.594 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:46:54.594 INFO:teuthology.orchestra.run.smithi123.stdout:Total 20 MB/s | 37 MB 00:01 2024-01-13T11:46:54.828 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-01-13T11:46:54.846 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-01-13T11:46:54.847 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-01-13T11:46:55.161 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-01-13T11:46:55.164 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-01-13T11:46:55.241 INFO:teuthology.orchestra.run.smithi059.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 14 MB/s | 8.6 MB 00:00 2024-01-13T11:46:55.762 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-01-13T11:46:55.802 INFO:teuthology.orchestra.run.smithi059.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 4.1 MB/s | 4.7 MB 00:01 2024-01-13T11:46:55.860 INFO:teuthology.orchestra.run.smithi059.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.0 MB/s | 4.9 MB 00:01 2024-01-13T11:46:55.886 INFO:teuthology.orchestra.run.smithi059.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.6 MB/s | 225 kB 00:00 2024-01-13T11:46:55.895 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-13T11:46:55.978 INFO:teuthology.orchestra.run.smithi059.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.1 MB/s | 3.7 MB 00:00 2024-01-13T11:46:56.026 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-13T11:46:56.187 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-13T11:46:56.194 INFO:teuthology.orchestra.run.smithi059.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 797 kB/s | 172 kB 00:00 2024-01-13T11:46:56.262 INFO:teuthology.orchestra.run.smithi059.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.7 MB/s | 645 kB 00:00 2024-01-13T11:46:56.429 INFO:teuthology.orchestra.run.smithi059.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 25 MB/s | 14 MB 00:00 2024-01-13T11:46:56.433 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:46:56.433 INFO:teuthology.orchestra.run.smithi059.stdout:Total 20 MB/s | 37 MB 00:01 2024-01-13T11:46:56.667 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-13T11:46:56.686 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-13T11:46:56.687 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-13T11:46:56.783 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-13T11:46:56.987 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-13T11:46:57.044 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-13T11:46:57.047 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-13T11:46:57.601 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-13T11:46:57.714 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-13T11:46:57.770 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-13T11:46:57.887 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-13T11:46:58.026 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-13T11:46:58.196 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-13T11:46:58.476 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-13T11:46:58.829 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-13T11:46:59.020 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-13T11:46:59.586 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-13T11:46:59.695 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-13T11:46:59.820 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-13T11:46:59.879 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-13T11:47:00.625 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-13T11:47:01.385 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-13T11:47:01.597 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-13T11:47:01.645 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-13T11:47:02.887 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-13T11:47:03.061 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-13T11:47:03.383 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-13T11:47:03.383 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-13T11:47:03.383 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-13T11:47:03.383 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-13T11:47:03.384 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-13T11:47:03.384 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-13T11:47:03.384 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-13T11:47:03.384 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-13T11:47:03.542 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-13T11:47:04.117 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-13T11:47:04.117 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:04.117 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-01-13T11:47:04.117 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-13T11:47:04.117 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-13T11:47:04.117 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-13T11:47:04.117 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-13T11:47:04.117 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-13T11:47:04.117 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-13T11:47:04.117 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-13T11:47:04.117 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-13T11:47:04.117 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:04.118 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:47:04.227 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-rook 2024-01-13T11:47:04.947 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-13T11:47:05.080 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:50 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:47:05.153 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-13T11:47:05.522 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-13T11:47:05.522 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-13T11:47:05.522 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-13T11:47:05.522 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-13T11:47:05.522 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-13T11:47:05.522 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-13T11:47:05.522 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-13T11:47:05.522 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-13T11:47:06.294 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-13T11:47:06.295 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:06.295 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-13T11:47:06.295 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-13T11:47:06.295 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-13T11:47:06.295 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-13T11:47:06.295 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-13T11:47:06.295 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-13T11:47:06.295 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-13T11:47:06.295 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-13T11:47:06.295 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-13T11:47:06.295 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:06.295 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:47:06.438 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-rook 2024-01-13T11:47:06.753 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:47:06.755 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:47:06.755 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-01-13T11:47:06.755 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:47:06.755 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-01-13T11:47:06.755 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-13T11:47:06.755 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-01-13T11:47:06.755 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-13T11:47:06.755 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-13T11:47:06.755 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-13T11:47:06.756 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-13T11:47:06.756 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-13T11:47:06.756 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-13T11:47:06.756 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-13T11:47:06.756 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-13T11:47:06.756 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-13T11:47:06.756 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-13T11:47:06.756 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-13T11:47:06.756 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-13T11:47:06.756 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:06.756 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-13T11:47:06.756 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:47:06.756 INFO:teuthology.orchestra.run.smithi123.stdout:Install 13 Packages 2024-01-13T11:47:06.757 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:06.757 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 1.9 M 2024-01-13T11:47:06.757 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 24 M 2024-01-13T11:47:06.757 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-01-13T11:47:07.332 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:58 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:47:07.574 INFO:teuthology.orchestra.run.smithi123.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 691 kB/s | 27 kB 00:00 2024-01-13T11:47:07.775 INFO:teuthology.orchestra.run.smithi123.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 83 kB/s | 20 kB 00:00 2024-01-13T11:47:07.791 INFO:teuthology.orchestra.run.smithi123.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 104 kB/s | 27 kB 00:00 2024-01-13T11:47:07.975 INFO:teuthology.orchestra.run.smithi123.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 316 kB/s | 126 kB 00:00 2024-01-13T11:47:08.017 INFO:teuthology.orchestra.run.smithi123.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 454 kB/s | 110 kB 00:00 2024-01-13T11:47:08.034 INFO:teuthology.orchestra.run.smithi123.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 640 kB/s | 155 kB 00:00 2024-01-13T11:47:08.109 INFO:teuthology.orchestra.run.smithi123.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 359 kB/s | 33 kB 00:00 2024-01-13T11:47:08.168 INFO:teuthology.orchestra.run.smithi123.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 122 kB/s | 16 kB 00:00 2024-01-13T11:47:08.193 INFO:teuthology.orchestra.run.smithi123.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 975 kB/s | 81 kB 00:00 2024-01-13T11:47:08.219 INFO:teuthology.orchestra.run.smithi123.stdout:(10/13): python3-requests-oauthlib-1.0.0-1.el8. 175 kB/s | 43 kB 00:00 2024-01-13T11:47:08.244 INFO:teuthology.orchestra.run.smithi123.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 1.2 MB/s | 64 kB 00:00 2024-01-13T11:47:08.411 INFO:teuthology.orchestra.run.smithi123.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 319 kB/s | 61 kB 00:00 2024-01-13T11:47:08.452 INFO:teuthology.orchestra.run.smithi123.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.1 MB/s | 1.2 MB 00:00 2024-01-13T11:47:08.453 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:47:08.453 INFO:teuthology.orchestra.run.smithi123.stdout:Total 1.1 MB/s | 1.9 MB 00:01 2024-01-13T11:47:08.510 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-01-13T11:47:08.526 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-01-13T11:47:08.526 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-01-13T11:47:08.612 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-01-13T11:47:08.615 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-01-13T11:47:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-01-13T11:47:09.155 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:47:09.157 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:47:09.157 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-01-13T11:47:09.157 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:47:09.157 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-13T11:47:09.157 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-13T11:47:09.157 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-01-13T11:47:09.157 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-13T11:47:09.157 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-13T11:47:09.157 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-13T11:47:09.157 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-13T11:47:09.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-13T11:47:09.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-13T11:47:09.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-13T11:47:09.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-13T11:47:09.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-13T11:47:09.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-13T11:47:09.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-13T11:47:09.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-13T11:47:09.158 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:09.158 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-13T11:47:09.158 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:47:09.158 INFO:teuthology.orchestra.run.smithi059.stdout:Install 13 Packages 2024-01-13T11:47:09.158 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:09.159 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 1.9 M 2024-01-13T11:47:09.159 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 24 M 2024-01-13T11:47:09.159 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-13T11:47:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-13T11:47:09.535 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-13T11:47:09.750 INFO:teuthology.orchestra.run.smithi059.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 670 kB/s | 27 kB 00:00 2024-01-13T11:47:09.804 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-13T11:47:09.909 INFO:teuthology.orchestra.run.smithi059.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 100 kB/s | 20 kB 00:00 2024-01-13T11:47:09.934 INFO:teuthology.orchestra.run.smithi059.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 119 kB/s | 27 kB 00:00 2024-01-13T11:47:10.082 INFO:teuthology.orchestra.run.smithi059.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 381 kB/s | 126 kB 00:00 2024-01-13T11:47:10.110 INFO:teuthology.orchestra.run.smithi059.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 549 kB/s | 110 kB 00:00 2024-01-13T11:47:10.143 INFO:teuthology.orchestra.run.smithi059.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 741 kB/s | 155 kB 00:00 2024-01-13T11:47:10.227 INFO:teuthology.orchestra.run.smithi059.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 295 kB/s | 43 kB 00:00 2024-01-13T11:47:10.252 INFO:teuthology.orchestra.run.smithi059.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 232 kB/s | 33 kB 00:00 2024-01-13T11:47:10.269 INFO:teuthology.orchestra.run.smithi059.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 135 kB/s | 16 kB 00:00 2024-01-13T11:47:10.361 INFO:teuthology.orchestra.run.smithi059.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 742 kB/s | 64 kB 00:00 2024-01-13T11:47:10.400 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-13T11:47:10.428 INFO:teuthology.orchestra.run.smithi059.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 405 kB/s | 81 kB 00:00 2024-01-13T11:47:10.445 INFO:teuthology.orchestra.run.smithi059.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 734 kB/s | 61 kB 00:00 2024-01-13T11:47:10.599 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-13T11:47:10.678 INFO:teuthology.orchestra.run.smithi059.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.7 MB/s | 1.2 MB 00:00 2024-01-13T11:47:10.681 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:47:10.681 INFO:teuthology.orchestra.run.smithi059.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2024-01-13T11:47:10.744 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-13T11:47:10.761 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-13T11:47:10.761 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-13T11:47:10.831 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-13T11:47:10.856 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-13T11:47:10.861 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-13T11:47:11.149 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-13T11:47:11.389 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-13T11:47:11.446 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-13T11:47:11.741 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-13T11:47:11.963 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-13T11:47:12.223 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-13T11:47:12.458 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-13T11:47:12.485 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-13T11:47:12.699 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-13T11:47:12.702 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-13T11:47:12.963 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-13T11:47:12.970 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-13T11:47:13.150 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-13T11:47:13.291 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-13T11:47:13.325 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-13T11:47:13.569 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-13T11:47:13.620 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-13T11:47:13.621 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-13T11:47:13.621 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-13T11:47:13.621 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-13T11:47:13.621 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-13T11:47:13.621 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-13T11:47:13.621 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-13T11:47:13.621 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-13T11:47:13.621 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-13T11:47:13.621 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-13T11:47:13.621 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-13T11:47:13.621 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-13T11:47:13.622 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-13T11:47:14.373 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-13T11:47:14.373 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:14.373 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-01-13T11:47:14.374 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-13T11:47:14.374 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-13T11:47:14.374 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-13T11:47:14.374 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-13T11:47:14.374 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-13T11:47:14.374 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-13T11:47:14.374 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-13T11:47:14.374 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-13T11:47:14.374 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-13T11:47:14.374 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-13T11:47:14.374 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-13T11:47:14.374 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-13T11:47:14.374 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-13T11:47:14.375 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:14.375 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:47:14.492 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-cephadm 2024-01-13T11:47:14.672 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-13T11:47:14.942 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-13T11:47:15.182 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-13T11:47:15.347 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:00 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:47:15.402 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-13T11:47:15.585 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-13T11:47:15.910 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-13T11:47:15.910 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-13T11:47:15.911 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-13T11:47:15.911 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-13T11:47:15.911 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-13T11:47:15.911 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-13T11:47:15.911 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-13T11:47:15.911 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-13T11:47:15.911 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-13T11:47:15.911 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-13T11:47:15.911 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-13T11:47:15.911 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-13T11:47:15.911 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-13T11:47:16.754 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-13T11:47:16.754 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:16.755 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-13T11:47:16.755 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-13T11:47:16.755 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-13T11:47:16.755 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-13T11:47:16.755 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-13T11:47:16.755 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-13T11:47:16.755 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-13T11:47:16.755 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-13T11:47:16.755 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-13T11:47:16.755 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-13T11:47:16.755 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-13T11:47:16.755 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-13T11:47:16.756 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-13T11:47:16.756 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-13T11:47:16.756 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:16.756 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:47:16.869 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-cephadm 2024-01-13T11:47:17.048 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:47:17.049 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:47:17.049 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-01-13T11:47:17.049 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:47:17.049 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-01-13T11:47:17.049 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-13T11:47:17.049 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-01-13T11:47:17.050 INFO:teuthology.orchestra.run.smithi123.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-13T11:47:17.050 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-13T11:47:17.050 INFO:teuthology.orchestra.run.smithi123.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-13T11:47:17.050 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-13T11:47:17.050 INFO:teuthology.orchestra.run.smithi123.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-13T11:47:17.050 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:17.050 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-13T11:47:17.050 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:47:17.050 INFO:teuthology.orchestra.run.smithi123.stdout:Install 6 Packages 2024-01-13T11:47:17.050 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:17.051 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 5.8 M 2024-01-13T11:47:17.051 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 24 M 2024-01-13T11:47:17.051 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-01-13T11:47:17.459 INFO:teuthology.orchestra.run.smithi123.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.9 MB/s | 89 kB 00:00 2024-01-13T11:47:17.593 INFO:teuthology.orchestra.run.smithi123.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 159 kB/s | 21 kB 00:00 2024-01-13T11:47:17.751 INFO:teuthology.orchestra.run.smithi123.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.2 MB/s | 355 kB 00:00 2024-01-13T11:47:17.762 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:08 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:47:17.810 INFO:teuthology.orchestra.run.smithi123.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 840 kB/s | 49 kB 00:00 2024-01-13T11:47:17.852 INFO:teuthology.orchestra.run.smithi123.stdout:(5/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.2 MB/s | 538 kB 00:00 2024-01-13T11:47:18.127 INFO:teuthology.orchestra.run.smithi123.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 6.7 MB/s | 4.8 MB 00:00 2024-01-13T11:47:18.127 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:47:18.128 INFO:teuthology.orchestra.run.smithi123.stdout:Total 5.4 MB/s | 5.8 MB 00:01 2024-01-13T11:47:18.180 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-01-13T11:47:18.189 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-01-13T11:47:18.190 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-01-13T11:47:18.290 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-01-13T11:47:18.293 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-01-13T11:47:18.565 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-01-13T11:47:18.866 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-13T11:47:19.144 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-13T11:47:19.596 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:47:19.597 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:47:19.597 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-01-13T11:47:19.597 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:47:19.598 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-13T11:47:19.598 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-13T11:47:19.598 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-01-13T11:47:19.598 INFO:teuthology.orchestra.run.smithi059.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-13T11:47:19.598 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-13T11:47:19.598 INFO:teuthology.orchestra.run.smithi059.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-13T11:47:19.598 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-13T11:47:19.598 INFO:teuthology.orchestra.run.smithi059.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-13T11:47:19.598 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:19.598 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-13T11:47:19.598 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:47:19.598 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2024-01-13T11:47:19.598 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:19.599 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 5.8 M 2024-01-13T11:47:19.599 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 24 M 2024-01-13T11:47:19.599 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-13T11:47:19.885 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.2 MB/s | 89 kB 00:00 2024-01-13T11:47:20.113 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-13T11:47:20.119 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 91 kB/s | 21 kB 00:00 2024-01-13T11:47:20.278 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.2 MB/s | 538 kB 00:00 2024-01-13T11:47:20.370 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.4 MB/s | 355 kB 00:00 2024-01-13T11:47:20.471 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-13T11:47:20.495 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): python3-babel-2.5.1-7.el8.noarch.rpm 7.4 MB/s | 4.8 MB 00:00 2024-01-13T11:47:20.572 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): python3-remoto-1.2.1-1.el8.noarch.rpm 166 kB/s | 49 kB 00:00 2024-01-13T11:47:20.574 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:47:20.574 INFO:teuthology.orchestra.run.smithi059.stdout:Total 6.0 MB/s | 5.8 MB 00:00 2024-01-13T11:47:20.629 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-13T11:47:20.640 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-13T11:47:20.640 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-13T11:47:20.750 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-13T11:47:20.754 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-13T11:47:20.946 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-13T11:47:21.071 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-13T11:47:21.174 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-13T11:47:21.340 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-13T11:47:21.470 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-13T11:47:21.470 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-13T11:47:21.470 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-13T11:47:21.470 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-13T11:47:21.470 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-13T11:47:21.470 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-13T11:47:21.632 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-13T11:47:22.174 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-13T11:47:22.174 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:22.174 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-01-13T11:47:22.174 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-13T11:47:22.174 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-13T11:47:22.174 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-13T11:47:22.174 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:22.174 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:47:22.306 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-fuse 2024-01-13T11:47:22.882 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-13T11:47:23.163 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:08 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:47:23.283 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-13T11:47:23.507 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-13T11:47:23.717 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-13T11:47:24.050 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-13T11:47:24.050 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-13T11:47:24.050 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-13T11:47:24.050 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-13T11:47:24.050 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-13T11:47:24.050 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-13T11:47:24.818 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-13T11:47:24.819 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:24.819 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-13T11:47:24.819 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-13T11:47:24.819 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-13T11:47:24.819 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-13T11:47:24.819 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:24.819 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:47:24.878 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:47:24.878 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:47:24.878 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-01-13T11:47:24.878 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:47:24.878 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-01-13T11:47:24.879 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-13T11:47:24.879 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-01-13T11:47:24.879 INFO:teuthology.orchestra.run.smithi123.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-13T11:47:24.879 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:24.879 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-13T11:47:24.879 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:47:24.879 INFO:teuthology.orchestra.run.smithi123.stdout:Install 2 Packages 2024-01-13T11:47:24.879 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:24.880 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 968 k 2024-01-13T11:47:24.880 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 3.2 M 2024-01-13T11:47:24.880 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-01-13T11:47:24.961 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-fuse 2024-01-13T11:47:24.979 INFO:teuthology.orchestra.run.smithi123.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2024-01-13T11:47:25.121 INFO:teuthology.orchestra.run.smithi123.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 422 kB/s | 83 kB 00:00 2024-01-13T11:47:25.122 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:47:25.122 INFO:teuthology.orchestra.run.smithi123.stdout:Total 3.9 MB/s | 968 kB 00:00 2024-01-13T11:47:25.132 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-01-13T11:47:25.152 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-01-13T11:47:25.152 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-01-13T11:47:25.238 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-01-13T11:47:25.241 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-01-13T11:47:25.421 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-01-13T11:47:25.691 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-13T11:47:25.865 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-13T11:47:25.874 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:16 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:47:26.253 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-13T11:47:26.254 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-13T11:47:26.992 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-13T11:47:26.993 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:26.993 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-01-13T11:47:26.993 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-13T11:47:26.993 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:26.993 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:47:27.175 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librados-devel 2024-01-13T11:47:27.715 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:47:27.715 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:47:27.716 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-01-13T11:47:27.716 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:47:27.716 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-13T11:47:27.716 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-13T11:47:27.716 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-01-13T11:47:27.716 INFO:teuthology.orchestra.run.smithi059.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-13T11:47:27.716 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:27.716 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-13T11:47:27.716 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:47:27.716 INFO:teuthology.orchestra.run.smithi059.stdout:Install 2 Packages 2024-01-13T11:47:27.716 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:27.717 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 968 k 2024-01-13T11:47:27.717 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 3.2 M 2024-01-13T11:47:27.717 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-13T11:47:27.989 INFO:teuthology.orchestra.run.smithi059.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 15 MB/s | 885 kB 00:00 2024-01-13T11:47:28.034 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:13 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:47:28.156 INFO:teuthology.orchestra.run.smithi059.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 372 kB/s | 83 kB 00:00 2024-01-13T11:47:28.157 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:47:28.157 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.1 MB/s | 968 kB 00:00 2024-01-13T11:47:28.168 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-13T11:47:28.191 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-13T11:47:28.191 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-13T11:47:28.285 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-13T11:47:28.288 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-13T11:47:28.512 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-13T11:47:28.814 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-13T11:47:29.000 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-13T11:47:29.414 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-13T11:47:29.414 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-13T11:47:29.746 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:47:29.747 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:47:29.747 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-01-13T11:47:29.747 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:47:29.747 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-01-13T11:47:29.747 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-13T11:47:29.747 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:29.747 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-13T11:47:29.747 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:47:29.748 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-01-13T11:47:29.748 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:29.748 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 131 k 2024-01-13T11:47:29.748 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 439 k 2024-01-13T11:47:29.748 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-01-13T11:47:29.807 INFO:teuthology.orchestra.run.smithi123.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 131 kB 00:00 2024-01-13T11:47:29.808 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:47:29.808 INFO:teuthology.orchestra.run.smithi123.stdout:Total 2.1 MB/s | 131 kB 00:00 2024-01-13T11:47:29.808 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-01-13T11:47:29.813 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-01-13T11:47:29.813 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-01-13T11:47:29.862 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-01-13T11:47:29.865 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-01-13T11:47:29.970 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-01-13T11:47:30.155 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-13T11:47:30.155 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:30.155 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-13T11:47:30.156 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-13T11:47:30.156 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:30.156 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:47:30.222 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:47:30.369 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados-devel 2024-01-13T11:47:30.510 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:47:31.233 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:47:31.233 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:31.233 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-01-13T11:47:31.233 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-13T11:47:31.233 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:31.233 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:47:31.263 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:22 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:47:32.427 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install libcephfs2 2024-01-13T11:47:33.127 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:47:33.128 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:47:33.128 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-01-13T11:47:33.128 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:47:33.128 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-13T11:47:33.128 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-13T11:47:33.128 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:33.128 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-13T11:47:33.128 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:47:33.128 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-01-13T11:47:33.128 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:33.129 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 131 k 2024-01-13T11:47:33.129 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 439 k 2024-01-13T11:47:33.129 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-13T11:47:33.285 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:18 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:47:33.813 INFO:teuthology.orchestra.run.smithi059.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 192 kB/s | 131 kB 00:00 2024-01-13T11:47:33.813 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:47:33.814 INFO:teuthology.orchestra.run.smithi059.stdout:Total 191 kB/s | 131 kB 00:00 2024-01-13T11:47:33.814 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-13T11:47:33.822 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-13T11:47:33.822 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-13T11:47:33.882 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-13T11:47:33.885 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-13T11:47:34.348 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-13T11:47:34.772 INFO:teuthology.orchestra.run.smithi123.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-13T11:47:34.924 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:47:35.010 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:47:35.011 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-01-13T11:47:35.011 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:47:35.122 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install libcephfs-devel 2024-01-13T11:47:35.244 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:47:35.969 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:20 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:47:35.970 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:47:35.970 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:35.970 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-13T11:47:35.970 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-13T11:47:35.970 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:35.970 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:47:36.258 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs2 2024-01-13T11:47:37.147 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:28 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:47:37.681 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:47:37.682 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:47:37.682 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-01-13T11:47:37.682 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:47:37.682 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-01-13T11:47:37.682 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-13T11:47:37.683 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:37.683 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-13T11:47:37.683 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:47:37.683 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-01-13T11:47:37.683 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:37.683 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 24 k 2024-01-13T11:47:37.683 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 94 k 2024-01-13T11:47:37.683 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-01-13T11:47:37.723 INFO:teuthology.orchestra.run.smithi123.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 623 kB/s | 24 kB 00:00 2024-01-13T11:47:37.723 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:47:37.723 INFO:teuthology.orchestra.run.smithi123.stdout:Total 604 kB/s | 24 kB 00:00 2024-01-13T11:47:37.724 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-01-13T11:47:37.726 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-01-13T11:47:37.726 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-01-13T11:47:37.728 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-01-13T11:47:37.731 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-01-13T11:47:37.778 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-01-13T11:47:37.962 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:47:38.230 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:47:38.761 INFO:teuthology.orchestra.run.smithi059.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-13T11:47:38.890 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:47:38.890 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:38.890 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-01-13T11:47:38.891 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-13T11:47:38.891 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:38.891 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:47:39.013 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:47:39.014 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-01-13T11:47:39.015 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:47:39.023 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librados2 2024-01-13T11:47:39.111 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs-devel 2024-01-13T11:47:39.878 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:24 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:47:39.999 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:31 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:47:41.359 INFO:teuthology.orchestra.run.smithi123.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-13T11:47:41.593 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:47:41.594 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-01-13T11:47:41.594 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:47:41.697 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librbd1 2024-01-13T11:47:41.832 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:47:41.833 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:47:41.833 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-01-13T11:47:41.833 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:47:41.833 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-13T11:47:41.834 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-13T11:47:41.834 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:41.834 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-13T11:47:41.834 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:47:41.834 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-01-13T11:47:41.834 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:41.834 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 24 k 2024-01-13T11:47:41.834 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 94 k 2024-01-13T11:47:41.834 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-13T11:47:41.874 INFO:teuthology.orchestra.run.smithi059.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 602 kB/s | 24 kB 00:00 2024-01-13T11:47:41.875 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:47:41.875 INFO:teuthology.orchestra.run.smithi059.stdout:Total 585 kB/s | 24 kB 00:00 2024-01-13T11:47:41.875 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-13T11:47:41.879 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-13T11:47:41.879 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-13T11:47:41.882 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-13T11:47:41.884 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-13T11:47:41.953 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-13T11:47:42.122 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:47:42.400 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:47:42.549 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:27 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:47:43.110 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:47:43.110 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:43.110 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-13T11:47:43.110 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-13T11:47:43.110 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:47:43.110 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:47:43.229 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados2 2024-01-13T11:47:44.030 INFO:teuthology.orchestra.run.smithi123.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-13T11:47:44.138 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:35 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:47:44.267 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:47:44.268 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-01-13T11:47:44.268 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:47:44.748 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rados 2024-01-13T11:47:45.593 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:30 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:47:45.726 INFO:teuthology.orchestra.run.smithi059.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-13T11:47:45.987 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:47:45.988 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-01-13T11:47:45.988 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:47:46.092 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librbd1 2024-01-13T11:47:46.990 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:37 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:47:47.075 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-13T11:47:47.316 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:47:47.317 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-01-13T11:47:47.317 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:47:47.421 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rgw 2024-01-13T11:47:48.268 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:33 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:47:48.568 INFO:teuthology.orchestra.run.smithi059.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-13T11:47:48.826 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:47:48.827 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-01-13T11:47:48.827 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:47:48.921 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rados 2024-01-13T11:47:49.773 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-13T11:47:49.823 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:40 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:47:50.007 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:47:50.008 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-01-13T11:47:50.008 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:47:50.112 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-cephfs 2024-01-13T11:47:50.967 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:35 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:47:51.401 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-13T11:47:51.659 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:47:51.660 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-01-13T11:47:51.660 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:47:51.765 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rgw 2024-01-13T11:47:52.431 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-13T11:47:52.649 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:43 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:47:52.668 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:47:52.669 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-01-13T11:47:52.669 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:47:52.777 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rbd 2024-01-13T11:47:53.641 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:38 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:47:54.230 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-13T11:47:54.484 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:47:54.486 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-01-13T11:47:54.486 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:47:54.583 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-cephfs 2024-01-13T11:47:55.116 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-13T11:47:55.350 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:47:55.352 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-01-13T11:47:55.352 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:47:55.467 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-fuse 2024-01-13T11:47:55.476 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:46 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:47:56.317 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:41 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:47:57.061 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-13T11:47:57.314 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:47:57.315 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-01-13T11:47:57.315 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:47:57.418 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rbd 2024-01-13T11:47:58.030 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:47:58.031 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:47:58.032 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-01-13T11:47:58.032 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:47:58.032 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-01-13T11:47:58.032 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-13T11:47:58.032 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:58.032 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-13T11:47:58.032 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:47:58.032 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-01-13T11:47:58.032 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:58.032 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 89 k 2024-01-13T11:47:58.032 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 298 k 2024-01-13T11:47:58.032 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-01-13T11:47:58.086 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.6 MB/s | 89 kB 00:00 2024-01-13T11:47:58.087 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:47:58.087 INFO:teuthology.orchestra.run.smithi123.stdout:Total 1.6 MB/s | 89 kB 00:00 2024-01-13T11:47:58.087 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-01-13T11:47:58.096 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-01-13T11:47:58.096 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-01-13T11:47:58.146 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-01-13T11:47:58.149 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-01-13T11:47:58.253 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-01-13T11:47:58.304 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:49 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:47:58.458 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:47:58.744 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:47:59.406 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:47:59.407 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:59.407 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-01-13T11:47:59.407 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-13T11:47:59.407 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:47:59.407 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:47:59.610 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-mirror 2024-01-13T11:47:59.879 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-13T11:48:00.130 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:48:00.131 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-01-13T11:48:00.132 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:48:00.224 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-fuse 2024-01-13T11:48:00.466 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:45 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:48:01.116 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:52 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:48:02.193 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:48:02.194 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:48:02.194 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-01-13T11:48:02.194 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:48:02.194 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-01-13T11:48:02.194 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-13T11:48:02.194 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:48:02.194 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-13T11:48:02.194 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:48:02.194 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-01-13T11:48:02.194 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:48:02.195 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 3.7 M 2024-01-13T11:48:02.195 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 18 M 2024-01-13T11:48:02.195 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-01-13T11:48:02.280 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 44 MB/s | 3.7 MB 00:00 2024-01-13T11:48:02.281 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:48:02.281 INFO:teuthology.orchestra.run.smithi123.stdout:Total 43 MB/s | 3.7 MB 00:00 2024-01-13T11:48:02.282 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-01-13T11:48:02.294 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-01-13T11:48:02.294 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-01-13T11:48:02.357 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-01-13T11:48:02.360 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-01-13T11:48:02.829 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-01-13T11:48:02.964 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:48:02.965 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:48:02.965 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-01-13T11:48:02.965 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:48:02.965 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-13T11:48:02.965 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-13T11:48:02.965 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:48:02.965 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-13T11:48:02.965 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:48:02.966 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-01-13T11:48:02.966 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:48:02.966 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 89 k 2024-01-13T11:48:02.966 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 298 k 2024-01-13T11:48:02.966 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-13T11:48:03.016 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 89 kB 00:00 2024-01-13T11:48:03.016 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:48:03.017 INFO:teuthology.orchestra.run.smithi059.stdout:Total 1.7 MB/s | 89 kB 00:00 2024-01-13T11:48:03.017 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-13T11:48:03.017 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:48:03.027 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-13T11:48:03.027 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-13T11:48:03.080 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-13T11:48:03.083 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-13T11:48:03.231 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-13T11:48:03.436 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:48:03.444 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:48:03.737 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:48:04.140 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:48:04.140 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:48:04.140 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-01-13T11:48:04.140 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-13T11:48:04.140 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:48:04.140 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:48:04.343 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-nbd 2024-01-13T11:48:04.466 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:48:04.466 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:48:04.466 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-13T11:48:04.466 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-13T11:48:04.466 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:48:04.466 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:48:04.707 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-mirror 2024-01-13T11:48:05.202 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:50 ago on Sat 13 Jan 2024 11:45:15 AM UTC. 2024-01-13T11:48:05.609 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:56 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:48:06.917 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-13T11:48:06.918 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:48:06.918 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-01-13T11:48:06.918 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:48:06.918 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-01-13T11:48:06.918 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-13T11:48:06.918 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:48:06.919 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-13T11:48:06.919 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-13T11:48:06.919 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-01-13T11:48:06.919 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:48:06.919 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 173 k 2024-01-13T11:48:06.919 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 649 k 2024-01-13T11:48:06.919 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-01-13T11:48:06.958 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.4 MB/s | 173 kB 00:00 2024-01-13T11:48:06.958 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:48:06.958 INFO:teuthology.orchestra.run.smithi123.stdout:Total 4.3 MB/s | 173 kB 00:00 2024-01-13T11:48:06.959 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-01-13T11:48:06.969 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-01-13T11:48:06.969 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-01-13T11:48:07.020 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-01-13T11:48:07.022 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-01-13T11:48:07.135 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-01-13T11:48:07.332 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:48:07.436 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:48:07.437 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:48:07.437 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-01-13T11:48:07.437 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:48:07.437 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-13T11:48:07.437 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-13T11:48:07.437 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:48:07.437 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-13T11:48:07.437 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:48:07.437 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-01-13T11:48:07.437 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:48:07.438 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 3.7 M 2024-01-13T11:48:07.438 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 18 M 2024-01-13T11:48:07.438 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-13T11:48:07.552 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 33 MB/s | 3.7 MB 00:00 2024-01-13T11:48:07.553 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:48:07.553 INFO:teuthology.orchestra.run.smithi059.stdout:Total 32 MB/s | 3.7 MB 00:00 2024-01-13T11:48:07.553 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-13T11:48:07.567 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-13T11:48:07.567 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-13T11:48:07.613 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:48:07.635 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-13T11:48:07.638 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-13T11:48:08.133 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-13T11:48:08.336 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:48:08.336 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:48:08.336 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-01-13T11:48:08.336 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-13T11:48:08.336 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:48:08.336 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-13T11:48:08.379 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:48:08.538 DEBUG:teuthology.parallel:result is None 2024-01-13T11:48:08.845 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:48:09.582 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:48:09.582 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:48:09.582 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-13T11:48:09.582 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-13T11:48:09.582 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:48:09.582 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:48:09.822 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-nbd 2024-01-13T11:48:10.720 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:01 ago on Sat 13 Jan 2024 11:45:09 AM UTC. 2024-01-13T11:48:12.579 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-13T11:48:12.580 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:48:12.580 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-01-13T11:48:12.580 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:48:12.580 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-13T11:48:12.580 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-13T11:48:12.580 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:48:12.580 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-13T11:48:12.580 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-13T11:48:12.580 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-01-13T11:48:12.581 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:48:12.581 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 173 k 2024-01-13T11:48:12.581 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 649 k 2024-01-13T11:48:12.581 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-13T11:48:12.621 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.2 MB/s | 173 kB 00:00 2024-01-13T11:48:12.621 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-13T11:48:12.622 INFO:teuthology.orchestra.run.smithi059.stdout:Total 4.1 MB/s | 173 kB 00:00 2024-01-13T11:48:12.622 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-13T11:48:12.633 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-13T11:48:12.634 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-13T11:48:12.687 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-13T11:48:12.690 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-13T11:48:12.815 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-13T11:48:13.001 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:48:13.326 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:48:14.082 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-13T11:48:14.082 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:48:14.082 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-13T11:48:14.082 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-13T11:48:14.082 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-13T11:48:14.082 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-13T11:48:14.274 DEBUG:teuthology.parallel:result is None 2024-01-13T11:48:14.274 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-13T11:48:14.275 INFO:teuthology.packaging:ref: None 2024-01-13T11:48:14.275 INFO:teuthology.packaging:tag: v16.2.4 2024-01-13T11:48:14.275 INFO:teuthology.packaging:branch: None 2024-01-13T11:48:14.275 INFO:teuthology.packaging:sha1: 445562ab4bc3ddfb386936119050695810860bcb 2024-01-13T11:48:14.275 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-13T11:48:14.549 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-13T11:48:14.573 INFO:teuthology.orchestra.run.smithi059.stdout:16.2.4-0.el8 2024-01-13T11:48:14.573 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-13T11:48:14.573 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-13T11:48:14.575 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-13T11:48:14.575 INFO:teuthology.packaging:ref: None 2024-01-13T11:48:14.575 INFO:teuthology.packaging:tag: v16.2.4 2024-01-13T11:48:14.575 INFO:teuthology.packaging:branch: None 2024-01-13T11:48:14.575 INFO:teuthology.packaging:sha1: 445562ab4bc3ddfb386936119050695810860bcb 2024-01-13T11:48:14.575 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-13T11:48:14.888 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-13T11:48:14.916 INFO:teuthology.orchestra.run.smithi123.stdout:16.2.4-0.el8 2024-01-13T11:48:14.916 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-13T11:48:14.916 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-13T11:48:14.918 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-13T11:48:14.918 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-13T11:48:14.918 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-13T11:48:14.951 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-01-13T11:48:14.951 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-13T11:48:14.991 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-13T11:48:14.991 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-13T11:48:14.991 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/daemon-helper 2024-01-13T11:48:15.022 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-13T11:48:15.091 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-01-13T11:48:15.091 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/daemon-helper 2024-01-13T11:48:15.124 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-13T11:48:15.198 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-13T11:48:15.198 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-13T11:48:15.198 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-13T11:48:15.227 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-13T11:48:15.295 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-01-13T11:48:15.295 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-13T11:48:15.326 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-13T11:48:15.396 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-13T11:48:15.396 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-13T11:48:15.396 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/stdin-killer 2024-01-13T11:48:15.425 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-13T11:48:15.493 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-01-13T11:48:15.493 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/stdin-killer 2024-01-13T11:48:15.522 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-13T11:48:15.591 INFO:teuthology.run_tasks:Running task print... 2024-01-13T11:48:15.601 INFO:teuthology.task.print:**** done install task... 2024-01-13T11:48:15.601 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-13T11:48:15.677 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '445562ab4bc3ddfb386936119050695810860bcb'} 2024-01-13T11:48:15.677 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-01-13T11:48:15.678 INFO:tasks.cephadm:Cluster fsid is a3276948-b209-11ee-95ac-87774f69a715 2024-01-13T11:48:15.678 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-13T11:48:15.678 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-13T11:48:15.678 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi059': '172.21.15.59', 'mon.smithi123': '172.21.15.123'} 2024-01-13T11:48:15.678 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-13T11:48:15.679 DEBUG:teuthology.orchestra.run.smithi059:> sudo hostname $(hostname -s) 2024-01-13T11:48:15.707 DEBUG:teuthology.orchestra.run.smithi123:> sudo hostname $(hostname -s) 2024-01-13T11:48:15.736 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-01-13T11:48:15.737 DEBUG:teuthology.orchestra.run.smithi059:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-13T11:48:15.995 INFO:teuthology.orchestra.run.smithi059.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 13 11:48 /home/ubuntu/cephtest/cephadm 2024-01-13T11:48:15.995 DEBUG:teuthology.orchestra.run.smithi123:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-13T11:48:16.131 INFO:teuthology.orchestra.run.smithi123.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 13 11:48 /home/ubuntu/cephtest/cephadm 2024-01-13T11:48:16.132 DEBUG:teuthology.orchestra.run.smithi059:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-13T11:48:16.154 DEBUG:teuthology.orchestra.run.smithi123:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-13T11:48:16.187 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph 2024-01-13T11:48:16.226 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /etc/ceph 2024-01-13T11:48:16.259 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 777 /etc/ceph 2024-01-13T11:48:16.297 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 777 /etc/ceph 2024-01-13T11:48:16.332 INFO:tasks.cephadm:Writing seed config... 2024-01-13T11:48:16.333 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-01-13T11:48:16.333 INFO:tasks.cephadm: override: [client] debug client = 20 2024-01-13T11:48:16.333 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-01-13T11:48:16.333 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-01-13T11:48:16.333 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-01-13T11:48:16.333 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-01-13T11:48:16.333 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-01-13T11:48:16.333 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-01-13T11:48:16.333 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-01-13T11:48:16.333 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-01-13T11:48:16.334 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-01-13T11:48:16.334 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-01-13T11:48:16.334 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-01-13T11:48:16.334 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-01-13T11:48:16.334 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-01-13T11:48:16.334 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-01-13T11:48:16.334 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-13T11:48:16.334 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-13T11:48:16.334 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-13T11:48:16.334 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-13T11:48:16.334 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-13T11:48:16.334 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-01-13T11:48:16.334 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-01-13T11:48:16.334 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-01-13T11:48:16.335 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-01-13T11:48:16.335 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-01-13T11:48:16.335 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-01-13T11:48:16.335 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-01-13T11:48:16.335 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-01-13T11:48:16.335 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-13T11:48:16.335 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-13T11:48:16.335 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-01-13T11:48:16.335 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-01-13T11:48:16.335 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-01-13T11:48:16.335 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-01-13T11:48:16.335 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-01-13T11:48:16.335 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-01-13T11:48:16.335 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-01-13T11:48:16.339 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-13T11:48:16.339 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-13T11:48:16.359 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a3276948-b209-11ee-95ac-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-01-13T11:48:16.359 DEBUG:teuthology.orchestra.run.smithi059:mon.smithi059> sudo journalctl -f -n 0 -u ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi059.service 2024-01-13T11:48:16.402 INFO:tasks.cephadm:Bootstrapping... 2024-01-13T11:48:16.402 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid a3276948-b209-11ee-95ac-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.59 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-13T11:48:16.432 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:-- Logs begin at Sat 2024-01-13 11:37:30 UTC. -- 2024-01-13T11:48:16.679 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-13T11:48:17.583 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: 4.6.1 2024-01-13T11:48:17.596 INFO:teuthology.orchestra.run.smithi059.stderr:Verifying podman|docker is present... 2024-01-13T11:48:17.596 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-13T11:48:17.620 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: 4.6.1 2024-01-13T11:48:17.632 INFO:teuthology.orchestra.run.smithi059.stderr:Verifying lvm2 is present... 2024-01-13T11:48:17.632 INFO:teuthology.orchestra.run.smithi059.stderr:Verifying time synchronization is in place... 2024-01-13T11:48:17.632 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl is-enabled chrony.service 2024-01-13T11:48:17.640 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-13T11:48:17.641 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl is-active chrony.service 2024-01-13T11:48:17.648 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: inactive 2024-01-13T11:48:17.648 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-13T11:48:17.655 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: enabled 2024-01-13T11:48:17.656 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl is-active chronyd.service 2024-01-13T11:48:17.662 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: active 2024-01-13T11:48:17.662 INFO:teuthology.orchestra.run.smithi059.stderr:Unit chronyd.service is enabled and running 2024-01-13T11:48:17.663 INFO:teuthology.orchestra.run.smithi059.stderr:Repeating the final host check... 2024-01-13T11:48:17.663 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-13T11:48:17.684 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: 4.6.1 2024-01-13T11:48:17.698 INFO:teuthology.orchestra.run.smithi059.stderr:podman|docker (/bin/podman) is present 2024-01-13T11:48:17.698 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl is present 2024-01-13T11:48:17.698 INFO:teuthology.orchestra.run.smithi059.stderr:lvcreate is present 2024-01-13T11:48:17.699 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl is-enabled chrony.service 2024-01-13T11:48:17.704 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-13T11:48:17.705 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl is-active chrony.service 2024-01-13T11:48:17.711 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: inactive 2024-01-13T11:48:17.711 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-13T11:48:17.717 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: enabled 2024-01-13T11:48:17.718 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl is-active chronyd.service 2024-01-13T11:48:17.724 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: active 2024-01-13T11:48:17.724 INFO:teuthology.orchestra.run.smithi059.stderr:Unit chronyd.service is enabled and running 2024-01-13T11:48:17.725 INFO:teuthology.orchestra.run.smithi059.stderr:Host looks OK 2024-01-13T11:48:17.725 INFO:teuthology.orchestra.run.smithi059.stderr:Cluster fsid: a3276948-b209-11ee-95ac-87774f69a715 2024-01-13T11:48:17.725 INFO:teuthology.orchestra.run.smithi059.stderr:Acquiring lock 139994179828200 on /run/cephadm/a3276948-b209-11ee-95ac-87774f69a715.lock 2024-01-13T11:48:17.725 INFO:teuthology.orchestra.run.smithi059.stderr:Lock 139994179828200 acquired on /run/cephadm/a3276948-b209-11ee-95ac-87774f69a715.lock 2024-01-13T11:48:17.725 INFO:teuthology.orchestra.run.smithi059.stderr:Verifying IP 172.21.15.59 port 3300 ... 2024-01-13T11:48:17.726 INFO:teuthology.orchestra.run.smithi059.stderr:Verifying IP 172.21.15.59 port 6789 ... 2024-01-13T11:48:17.726 INFO:teuthology.orchestra.run.smithi059.stderr:Base mon IP is 172.21.15.59, final addrv is [v2:172.21.15.59:3300,v1:172.21.15.59:6789] 2024-01-13T11:48:17.726 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /sbin/ip route ls 2024-01-13T11:48:17.728 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.59 metric 100 2024-01-13T11:48:17.728 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.59 metric 100 2024-01-13T11:48:17.729 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /sbin/ip -6 route ls 2024-01-13T11:48:17.731 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-01-13T11:48:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-01-13T11:48:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1795sec hoplimit 64 pref medium 2024-01-13T11:48:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /sbin/ip -6 addr ls 2024-01-13T11:48:17.735 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-13T11:48:17.735 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-01-13T11:48:17.735 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-13T11:48:17.735 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-01-13T11:48:17.735 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:192a/64 scope link 2024-01-13T11:48:17.735 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-13T11:48:17.736 INFO:teuthology.orchestra.run.smithi059.stderr:Mon IP 172.21.15.59 is in CIDR network 172.21.0.0/20 2024-01-13T11:48:17.736 INFO:teuthology.orchestra.run.smithi059.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-13T11:48:17.737 INFO:teuthology.orchestra.run.smithi059.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-01-13T11:48:17.737 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-01-13T11:48:17.839 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-01-13T11:48:24.523 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: Getting image source signatures 2024-01-13T11:48:24.539 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-01-13T11:48:24.539 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-01-13T11:48:41.536 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-13T11:48:41.602 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: Writing manifest to image destination 2024-01-13T11:48:41.982 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-13T11:48:41.993 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-01-13T11:48:42.544 INFO:teuthology.orchestra.run.smithi059.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-13T11:48:42.839 INFO:teuthology.orchestra.run.smithi059.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-13T11:48:42.839 INFO:teuthology.orchestra.run.smithi059.stderr:Extracting ceph user uid/gid from container image... 2024-01-13T11:48:42.840 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-01-13T11:48:43.159 INFO:teuthology.orchestra.run.smithi059.stderr:stat: 167 167 2024-01-13T11:48:43.477 INFO:teuthology.orchestra.run.smithi059.stderr:Creating initial keys... 2024-01-13T11:48:43.477 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-13T11:48:43.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-authtool: AQCbeKJla8MKLxAATUiUpqoHiydHFS0WgJybnQ== 2024-01-13T11:48:44.159 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-13T11:48:44.434 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-authtool: AQCceKJlEHowGRAACQCRLrc+0gbHYUr8Ifs2zg== 2024-01-13T11:48:44.756 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-13T11:48:45.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-authtool: AQCdeKJl98DDChAAutH4aYNTw5N9m+/PCtjesw== 2024-01-13T11:48:45.514 INFO:teuthology.orchestra.run.smithi059.stderr:Creating initial monmap... 2024-01-13T11:48:45.514 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpvlp0ahbn:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid a3276948-b209-11ee-95ac-87774f69a715 --addv smithi059 [v2:172.21.15.59:3300,v1:172.21.15.59:6789] /tmp/monmap 2024-01-13T11:48:45.820 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-13T11:48:45.820 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to a3276948-b209-11ee-95ac-87774f69a715 2024-01-13T11:48:45.820 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-13T11:48:46.148 INFO:teuthology.orchestra.run.smithi059.stderr:monmaptool for smithi059 [v2:172.21.15.59:3300,v1:172.21.15.59:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-13T11:48:46.148 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/monmaptool: set fsid to a3276948-b209-11ee-95ac-87774f69a715 2024-01-13T11:48:46.148 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-13T11:48:46.148 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-13T11:48:46.148 INFO:teuthology.orchestra.run.smithi059.stderr:Creating mon... 2024-01-13T11:48:46.149 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/mon.smithi059:/var/lib/ceph/mon/ceph-smithi059:z -v /tmp/ceph-tmprwcm9xme:/tmp/keyring:z -v /tmp/ceph-tmpvlp0ahbn:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi059 --fsid a3276948-b209-11ee-95ac-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-01-13T11:48:46.433 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.413+0000 7ff673d0c700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-13T11:48:46.433 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 1 imported monmap: 2024-01-13T11:48:46.434 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: epoch 0 2024-01-13T11:48:46.434 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: fsid a3276948-b209-11ee-95ac-87774f69a715 2024-01-13T11:48:46.434 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: last_changed 2024-01-13T11:48:45.820605+0000 2024-01-13T11:48:46.434 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: created 2024-01-13T11:48:45.820605+0000 2024-01-13T11:48:46.434 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-01-13T11:48:46.434 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-01-13T11:48:46.434 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.smithi059 2024-01-13T11:48:46.434 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2024-01-13T11:48:46.434 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 0 /usr/bin/ceph-mon: set fsid to a3276948-b209-11ee-95ac-87774f69a715 2024-01-13T11:48:46.434 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: RocksDB version: 6.8.1 2024-01-13T11:48:46.434 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2024-01-13T11:48:46.434 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-13T11:48:46.434 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Compile date May 13 2021 2024-01-13T11:48:46.434 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: DB SUMMARY 2024-01-13T11:48:46.435 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2024-01-13T11:48:46.435 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi059/store.db dir, Total Num: 0, files: 2024-01-13T11:48:46.435 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2024-01-13T11:48:46.435 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi059/store.db: 2024-01-13T11:48:46.435 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2024-01-13T11:48:46.435 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.error_if_exists: 0 2024-01-13T11:48:46.435 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.create_if_missing: 1 2024-01-13T11:48:46.435 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.paranoid_checks: 1 2024-01-13T11:48:46.435 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.env: 0x55f8e98551c0 2024-01-13T11:48:46.435 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.fs: Posix File System 2024-01-13T11:48:46.435 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.info_log: 0x55f8ea188ec0 2024-01-13T11:48:46.435 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-13T11:48:46.435 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.statistics: (nil) 2024-01-13T11:48:46.435 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.use_fsync: 0 2024-01-13T11:48:46.436 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.max_log_file_size: 0 2024-01-13T11:48:46.436 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-13T11:48:46.436 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-13T11:48:46.436 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-13T11:48:46.436 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-13T11:48:46.436 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.allow_fallocate: 1 2024-01-13T11:48:46.436 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-13T11:48:46.436 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-13T11:48:46.436 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.use_direct_reads: 0 2024-01-13T11:48:46.436 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-13T11:48:46.436 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-13T11:48:46.436 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.db_log_dir: 2024-01-13T11:48:46.436 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi059/store.db 2024-01-13T11:48:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-13T11:48:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.max_subcompactions: 1 2024-01-13T11:48:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.max_background_flushes: -1 2024-01-13T11:48:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-13T11:48:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-13T11:48:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-13T11:48:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-13T11:48:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-13T11:48:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-13T11:48:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-13T11:48:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.write_buffer_manager: 0x55f8ea18f530 2024-01-13T11:48:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-13T11:48:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-13T11:48:46.438 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-13T11:48:46.438 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-13T11:48:46.438 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-13T11:48:46.438 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-13T11:48:46.438 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-13T11:48:46.438 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-13T11:48:46.438 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-13T11:48:46.438 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.unordered_write: 0 2024-01-13T11:48:46.438 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-13T11:48:46.438 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-13T11:48:46.438 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-13T11:48:46.438 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-13T11:48:46.438 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.row_cache: None 2024-01-13T11:48:46.438 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.wal_filter: None 2024-01-13T11:48:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-13T11:48:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-13T11:48:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.preserve_deletes: 0 2024-01-13T11:48:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.two_write_queues: 0 2024-01-13T11:48:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-13T11:48:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.atomic_flush: 0 2024-01-13T11:48:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-13T11:48:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-13T11:48:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-13T11:48:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.log_readahead_size: 0 2024-01-13T11:48:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-13T11:48:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.max_background_jobs: 2 2024-01-13T11:48:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.max_background_compactions: -1 2024-01-13T11:48:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-13T11:48:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-13T11:48:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-13T11:48:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-13T11:48:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-13T11:48:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-13T11:48:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-13T11:48:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-13T11:48:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.max_open_files: -1 2024-01-13T11:48:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-13T11:48:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-13T11:48:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-13T11:48:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-13T11:48:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Compression algorithms supported: 2024-01-13T11:48:46.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-13T11:48:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: kZSTD supported: 0 2024-01-13T11:48:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: kXpressCompression supported: 0 2024-01-13T11:48:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-13T11:48:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: kLZ4Compression supported: 1 2024-01-13T11:48:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: kBZip2Compression supported: 0 2024-01-13T11:48:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: kZlibCompression supported: 1 2024-01-13T11:48:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: kSnappyCompression supported: 1 2024-01-13T11:48:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.414+0000 7ff673d0c700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-13T11:48:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-01-13T11:48:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2024-01-13T11:48:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi059/store.db/MANIFEST-000001 2024-01-13T11:48:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2024-01-13T11:48:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-13T11:48:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2024-01-13T11:48:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-13T11:48:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.merge_operator: 2024-01-13T11:48:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.compaction_filter: None 2024-01-13T11:48:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-13T11:48:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-13T11:48:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-13T11:48:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f8ea0a0120) 2024-01-13T11:48:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-01-13T11:48:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-13T11:48:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-13T11:48:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-01-13T11:48:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: index_type: 0 2024-01-13T11:48:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-01-13T11:48:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-01-13T11:48:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-01-13T11:48:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-01-13T11:48:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: checksum: 1 2024-01-13T11:48:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-01-13T11:48:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: block_cache: 0x55f8ea0d6d10 2024-01-13T11:48:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-01-13T11:48:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-01-13T11:48:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-01-13T11:48:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-01-13T11:48:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-01-13T11:48:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-01-13T11:48:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-01-13T11:48:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-01-13T11:48:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-01-13T11:48:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-01-13T11:48:46.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-01-13T11:48:46.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-01-13T11:48:46.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-01-13T11:48:46.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-01-13T11:48:46.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-01-13T11:48:46.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-13T11:48:46.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-01-13T11:48:46.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-01-13T11:48:46.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-01-13T11:48:46.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: format_version: 2 2024-01-13T11:48:46.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-01-13T11:48:46.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: block_align: 0 2024-01-13T11:48:46.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2024-01-13T11:48:46.445 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-13T11:48:46.445 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-13T11:48:46.445 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.compression: NoCompression 2024-01-13T11:48:46.445 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-13T11:48:46.445 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-13T11:48:46.445 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-13T11:48:46.445 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.num_levels: 7 2024-01-13T11:48:46.445 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-13T11:48:46.445 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-13T11:48:46.445 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-13T11:48:46.445 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-13T11:48:46.445 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-13T11:48:46.445 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-13T11:48:46.445 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-13T11:48:46.446 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-13T11:48:46.446 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-13T11:48:46.446 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-13T11:48:46.446 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-13T11:48:46.446 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-13T11:48:46.446 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-13T11:48:46.446 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-13T11:48:46.446 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-13T11:48:46.446 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-13T11:48:46.446 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-13T11:48:46.446 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-13T11:48:46.446 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-13T11:48:46.446 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-13T11:48:46.446 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-13T11:48:46.446 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-13T11:48:46.447 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-13T11:48:46.447 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-13T11:48:46.447 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-13T11:48:46.447 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-13T11:48:46.447 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-13T11:48:46.447 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-13T11:48:46.447 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-13T11:48:46.447 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-13T11:48:46.447 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-13T11:48:46.447 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-13T11:48:46.447 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-13T11:48:46.447 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-13T11:48:46.447 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-13T11:48:46.447 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-13T11:48:46.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-13T11:48:46.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-13T11:48:46.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-13T11:48:46.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-13T11:48:46.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-13T11:48:46.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-13T11:48:46.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-13T11:48:46.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-13T11:48:46.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-13T11:48:46.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-13T11:48:46.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-13T11:48:46.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.table_properties_collectors: 2024-01-13T11:48:46.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.inplace_update_support: 0 2024-01-13T11:48:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-13T11:48:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-13T11:48:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-13T11:48:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-13T11:48:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.bloom_locality: 0 2024-01-13T11:48:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.max_successive_merges: 0 2024-01-13T11:48:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-13T11:48:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-13T11:48:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-13T11:48:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-13T11:48:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.ttl: 2592000 2024-01-13T11:48:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.415+0000 7ff673d0c700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-13T11:48:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.416+0000 7ff673d0c700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi059/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-13T11:48:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2024-01-13T11:48:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.416+0000 7ff673d0c700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-13T11:48:46.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2024-01-13T11:48:46.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.416+0000 7ff673d0c700 4 rocksdb: DB pointer 0x55f8ea19b800 2024-01-13T11:48:46.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.416+0000 7ff65cba5700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-13T11:48:46.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.416+0000 7ff65cba5700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-13T11:48:46.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-01-13T11:48:46.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T11:48:46.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-13T11:48:46.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-13T11:48:46.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T11:48:46.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-13T11:48:46.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-13T11:48:46.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T11:48:46.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2024-01-13T11:48:46.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T11:48:46.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:48:46.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:48:46.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T11:48:46.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T11:48:46.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2024-01-13T11:48:46.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T11:48:46.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:48:46.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:48:46.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T11:48:46.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T11:48:46.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T11:48:46.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T11:48:46.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T11:48:46.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-13T11:48:46.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:48:46.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:48:46.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T11:48:46.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2024-01-13T11:48:46.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-13T11:48:46.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2024-01-13T11:48:46.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T11:48:46.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:48:46.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:48:46.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T11:48:46.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T11:48:46.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2024-01-13T11:48:46.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T11:48:46.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:48:46.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:48:46.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T11:48:46.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T11:48:46.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T11:48:46.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T11:48:46.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T11:48:46.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-13T11:48:46.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:48:46.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:48:46.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T11:48:46.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2024-01-13T11:48:46.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-13T11:48:46.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2024-01-13T11:48:46.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.418+0000 7ff673d0c700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-01-13T11:48:46.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.418+0000 7ff673d0c700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-01-13T11:48:46.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:48:46.418+0000 7ff673d0c700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi059 for mon.smithi059 2024-01-13T11:48:46.744 INFO:teuthology.orchestra.run.smithi059.stderr:create mon.smithi059 on 2024-01-13T11:48:46.744 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/a3276948-b209-11ee-95ac-87774f69a715 2024-01-13T11:48:46.749 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl disable ceph.target 2024-01-13T11:48:46.756 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-01-13T11:48:46.847 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl enable ceph.target 2024-01-13T11:48:46.855 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-13T11:48:46.942 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl start ceph.target 2024-01-13T11:48:46.949 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl enable ceph-a3276948-b209-11ee-95ac-87774f69a715.target 2024-01-13T11:48:46.955 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a3276948-b209-11ee-95ac-87774f69a715.target → /etc/systemd/system/ceph-a3276948-b209-11ee-95ac-87774f69a715.target. 2024-01-13T11:48:46.955 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-a3276948-b209-11ee-95ac-87774f69a715.target → /etc/systemd/system/ceph-a3276948-b209-11ee-95ac-87774f69a715.target. 2024-01-13T11:48:47.041 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl start ceph-a3276948-b209-11ee-95ac-87774f69a715.target 2024-01-13T11:48:47.048 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl daemon-reload 2024-01-13T11:48:47.134 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl stop ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi059 2024-01-13T11:48:47.162 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl reset-failed ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi059 2024-01-13T11:48:47.167 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Failed to reset failed state of unit ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi059.service: Unit ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi059.service not loaded. 2024-01-13T11:48:47.168 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl enable ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi059 2024-01-13T11:48:47.175 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Created symlink /etc/systemd/system/ceph-a3276948-b209-11ee-95ac-87774f69a715.target.wants/ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi059.service → /etc/systemd/system/ceph-a3276948-b209-11ee-95ac-87774f69a715@.service. 2024-01-13T11:48:47.266 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl start ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi059 2024-01-13T11:48:47.557 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-13T11:48:47.565 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: disabled 2024-01-13T11:48:47.566 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl is-active firewalld.service 2024-01-13T11:48:47.573 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: inactive 2024-01-13T11:48:47.574 INFO:teuthology.orchestra.run.smithi059.stderr:firewalld.service is not enabled 2024-01-13T11:48:47.574 INFO:teuthology.orchestra.run.smithi059.stderr:Not possible to enable service . firewalld.service is not available 2024-01-13T11:48:47.574 INFO:teuthology.orchestra.run.smithi059.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-13T11:48:47.575 INFO:teuthology.orchestra.run.smithi059.stderr:Waiting for mon to start... 2024-01-13T11:48:47.575 INFO:teuthology.orchestra.run.smithi059.stderr:Waiting for mon... 2024-01-13T11:48:47.575 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/mon.smithi059:/var/lib/ceph/mon/ceph-smithi059:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-01-13T11:48:47.816 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:47 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38356]: cluster 2024-01-13T11:48:47.590229+0000 mon.smithi059 (mon.0) 1 : cluster [INF] mon.smithi059 is new leader, mons smithi059 in quorum (ranks 0) 2024-01-13T11:48:48.003 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.013+0000 7f7bbb066700 1 Processor -- start 2024-01-13T11:48:48.006 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.013+0000 7f7bbb066700 1 -- start start 2024-01-13T11:48:48.006 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.013+0000 7f7bbb066700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7bb40703f0 0x7f7bb40fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:48:48.006 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.013+0000 7f7bbb066700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb40fee10 con 0x7f7bb40703f0 2024-01-13T11:48:48.006 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.014+0000 7f7bb8e02700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7bb40703f0 0x7f7bb40fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:48:48.006 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.014+0000 7f7bb8e02700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7bb40703f0 0x7f7bb40fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59566/0 (socket says 172.21.15.59:59566) 2024-01-13T11:48:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.014+0000 7f7bb8e02700 1 -- 172.21.15.59:0/2217927836 learned_addr learned my addr 172.21.15.59:0/2217927836 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:48:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.014+0000 7f7bb8e02700 1 -- 172.21.15.59:0/2217927836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb40fef50 con 0x7f7bb40703f0 2024-01-13T11:48:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.014+0000 7f7bb8e02700 1 --2- 172.21.15.59:0/2217927836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7bb40703f0 0x7f7bb40fe920 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f7ba800b5f0 tx=0x7f7ba8006ee0).ready entity=mon.0 client_cookie=4383701dc65ecce0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:48:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.014+0000 7f7bb37fe700 1 -- 172.21.15.59:0/2217927836 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ba800d600 con 0x7f7bb40703f0 2024-01-13T11:48:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.014+0000 7f7bb37fe700 1 -- 172.21.15.59:0/2217927836 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba800d760 con 0x7f7bb40703f0 2024-01-13T11:48:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.014+0000 7f7bb37fe700 1 -- 172.21.15.59:0/2217927836 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ba800da30 con 0x7f7bb40703f0 2024-01-13T11:48:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.015+0000 7f7bbb066700 1 -- 172.21.15.59:0/2217927836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7bb40703f0 msgr2=0x7f7bb40fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:48:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.015+0000 7f7bbb066700 1 --2- 172.21.15.59:0/2217927836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7bb40703f0 0x7f7bb40fe920 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f7ba800b5f0 tx=0x7f7ba8006ee0).stop 2024-01-13T11:48:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.015+0000 7f7bbb066700 1 -- 172.21.15.59:0/2217927836 shutdown_connections 2024-01-13T11:48:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.015+0000 7f7bbb066700 1 --2- 172.21.15.59:0/2217927836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7bb40703f0 0x7f7bb40fe920 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:48:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.015+0000 7f7bbb066700 1 -- 172.21.15.59:0/2217927836 >> 172.21.15.59:0/2217927836 conn(0x7f7bb40f63c0 msgr2=0x7f7bb40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:48:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.015+0000 7f7bbb066700 1 -- 172.21.15.59:0/2217927836 shutdown_connections 2024-01-13T11:48:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.015+0000 7f7bbb066700 1 -- 172.21.15.59:0/2217927836 wait complete. 2024-01-13T11:48:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.015+0000 7f7bbb066700 1 Processor -- start 2024-01-13T11:48:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.016+0000 7f7bbb066700 1 -- start start 2024-01-13T11:48:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.016+0000 7f7bbb066700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7bb40703f0 0x7f7bb40fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:48:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.016+0000 7f7bbb066700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb40fa820 con 0x7f7bb40703f0 2024-01-13T11:48:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.016+0000 7f7bb8e02700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7bb40703f0 0x7f7bb40fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:48:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.016+0000 7f7bb8e02700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7bb40703f0 0x7f7bb40fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59574/0 (socket says 172.21.15.59:59574) 2024-01-13T11:48:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.016+0000 7f7bb8e02700 1 -- 172.21.15.59:0/3371677316 learned_addr learned my addr 172.21.15.59:0/3371677316 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:48:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.016+0000 7f7bb8e02700 1 -- 172.21.15.59:0/3371677316 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba800b200 con 0x7f7bb40703f0 2024-01-13T11:48:48.009 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.016+0000 7f7bb8e02700 1 --2- 172.21.15.59:0/3371677316 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7bb40703f0 0x7f7bb40fc3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f7ba8006c70 tx=0x7f7ba8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:48:48.009 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.017+0000 7f7bb17fa700 1 -- 172.21.15.59:0/3371677316 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ba8011b40 con 0x7f7bb40703f0 2024-01-13T11:48:48.009 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.017+0000 7f7bb17fa700 1 -- 172.21.15.59:0/3371677316 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba8011ca0 con 0x7f7bb40703f0 2024-01-13T11:48:48.009 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.017+0000 7f7bbb066700 1 -- 172.21.15.59:0/3371677316 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb40fa9c0 con 0x7f7bb40703f0 2024-01-13T11:48:48.009 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.017+0000 7f7bb17fa700 1 -- 172.21.15.59:0/3371677316 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ba801d430 con 0x7f7bb40703f0 2024-01-13T11:48:48.009 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.017+0000 7f7bbb066700 1 -- 172.21.15.59:0/3371677316 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb40fadd0 con 0x7f7bb40703f0 2024-01-13T11:48:48.009 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.017+0000 7f7bb17fa700 1 -- 172.21.15.59:0/3371677316 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f7ba800f070 con 0x7f7bb40703f0 2024-01-13T11:48:48.009 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.017+0000 7f7bb17fa700 1 -- 172.21.15.59:0/3371677316 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f7ba8027460 con 0x7f7bb40703f0 2024-01-13T11:48:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.018+0000 7f7bbb066700 1 -- 172.21.15.59:0/3371677316 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb4059100 con 0x7f7bb40703f0 2024-01-13T11:48:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.020+0000 7f7bb17fa700 1 -- 172.21.15.59:0/3371677316 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f7ba8016070 con 0x7f7bb40703f0 2024-01-13T11:48:48.081 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.090+0000 7f7bbb066700 1 -- 172.21.15.59:0/3371677316 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f7bb4103df0 con 0x7f7bb40703f0 2024-01-13T11:48:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.090+0000 7f7bb17fa700 1 -- 172.21.15.59:0/3371677316 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f7ba8036070 con 0x7f7bb40703f0 2024-01-13T11:48:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: cluster: 2024-01-13T11:48:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: id: a3276948-b209-11ee-95ac-87774f69a715 2024-01-13T11:48:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-01-13T11:48:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: services: 2024-01-13T11:48:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi059 (age 0.483843s) 2024-01-13T11:48:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mgr: no daemons active 2024-01-13T11:48:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-01-13T11:48:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: data: 2024-01-13T11:48:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-01-13T11:48:48.083 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-01-13T11:48:48.083 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-01-13T11:48:48.083 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: pgs: 2024-01-13T11:48:48.083 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.083 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.091+0000 7f7bbb066700 1 -- 172.21.15.59:0/3371677316 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7bb40703f0 msgr2=0x7f7bb40fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:48:48.083 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.091+0000 7f7bbb066700 1 --2- 172.21.15.59:0/3371677316 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7bb40703f0 0x7f7bb40fc3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f7ba8006c70 tx=0x7f7ba8016000).stop 2024-01-13T11:48:48.083 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.091+0000 7f7bbb066700 1 -- 172.21.15.59:0/3371677316 shutdown_connections 2024-01-13T11:48:48.083 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.091+0000 7f7bbb066700 1 --2- 172.21.15.59:0/3371677316 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7bb40703f0 0x7f7bb40fc3a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:48:48.083 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.091+0000 7f7bbb066700 1 -- 172.21.15.59:0/3371677316 >> 172.21.15.59:0/3371677316 conn(0x7f7bb40f63c0 msgr2=0x7f7bb40f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:48:48.083 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.091+0000 7f7bbb066700 1 -- 172.21.15.59:0/3371677316 shutdown_connections 2024-01-13T11:48:48.083 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:48.091+0000 7f7bbb066700 1 -- 172.21.15.59:0/3371677316 wait complete. 2024-01-13T11:48:48.621 INFO:teuthology.orchestra.run.smithi059.stderr:mon is available 2024-01-13T11:48:48.622 INFO:teuthology.orchestra.run.smithi059.stderr:Assimilating anything we can from ceph.conf... 2024-01-13T11:48:48.622 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z -v /var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/mon.smithi059:/var/lib/ceph/mon/ceph-smithi059:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi059/config 2024-01-13T11:48:48.771 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:48 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38356]: cluster 2024-01-13T11:48:47.606855+0000 mon.smithi059 (mon.0) 2 : cluster [INF] mon.smithi059 is new leader, mons smithi059 in quorum (ranks 0) 2024-01-13T11:48:48.771 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:48 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38356]: cluster 2024-01-13T11:48:47.607088+0000 mon.smithi059 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi059=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} 2024-01-13T11:48:48.771 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:48 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38356]: cluster 2024-01-13T11:48:47.610914+0000 mon.smithi059 (mon.0) 4 : cluster [DBG] fsmap 2024-01-13T11:48:48.772 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:48 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38356]: cluster 2024-01-13T11:48:47.614710+0000 mon.smithi059 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-13T11:48:48.772 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:48 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38356]: cluster 2024-01-13T11:48:47.615175+0000 mon.smithi059 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-01-13T11:48:48.772 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:48 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38356]: audit 2024-01-13T11:48:48.090665+0000 mon.smithi059 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.59:0/3371677316' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-13T11:48:49.029 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.034+0000 7f01b21c2700 1 Processor -- start 2024-01-13T11:48:49.029 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.034+0000 7f01b21c2700 1 -- start start 2024-01-13T11:48:49.029 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.034+0000 7f01b21c2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ac0fd380 0x7f01ac0fd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:48:49.029 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.034+0000 7f01b21c2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac0fdc90 con 0x7f01ac0fd380 2024-01-13T11:48:49.029 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.034+0000 7f01ab7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ac0fd380 0x7f01ac0fd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:48:49.030 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.034+0000 7f01ab7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ac0fd380 0x7f01ac0fd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59586/0 (socket says 172.21.15.59:59586) 2024-01-13T11:48:49.030 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.034+0000 7f01ab7fe700 1 -- 172.21.15.59:0/453457339 learned_addr learned my addr 172.21.15.59:0/453457339 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:48:49.030 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.034+0000 7f01ab7fe700 1 -- 172.21.15.59:0/453457339 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ac0fddd0 con 0x7f01ac0fd380 2024-01-13T11:48:49.030 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.034+0000 7f01ab7fe700 1 --2- 172.21.15.59:0/453457339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ac0fd380 0x7f01ac0fd7a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f019c00b5f0 tx=0x7f019c006ee0).ready entity=mon.0 client_cookie=f9e8a20ec83057d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:48:49.030 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.035+0000 7f01aa7fc700 1 -- 172.21.15.59:0/453457339 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f019c00d600 con 0x7f01ac0fd380 2024-01-13T11:48:49.030 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.035+0000 7f01aa7fc700 1 -- 172.21.15.59:0/453457339 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c00d760 con 0x7f01ac0fd380 2024-01-13T11:48:49.030 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.035+0000 7f01aa7fc700 1 -- 172.21.15.59:0/453457339 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f019c00da30 con 0x7f01ac0fd380 2024-01-13T11:48:49.030 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.035+0000 7f01b21c2700 1 -- 172.21.15.59:0/453457339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ac0fd380 msgr2=0x7f01ac0fd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:48:49.030 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.035+0000 7f01b21c2700 1 --2- 172.21.15.59:0/453457339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ac0fd380 0x7f01ac0fd7a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f019c00b5f0 tx=0x7f019c006ee0).stop 2024-01-13T11:48:49.030 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.035+0000 7f01b21c2700 1 -- 172.21.15.59:0/453457339 shutdown_connections 2024-01-13T11:48:49.031 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.035+0000 7f01b21c2700 1 --2- 172.21.15.59:0/453457339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ac0fd380 0x7f01ac0fd7a0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:48:49.031 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.035+0000 7f01b21c2700 1 -- 172.21.15.59:0/453457339 >> 172.21.15.59:0/453457339 conn(0x7f01ac0f5b40 msgr2=0x7f01ac0f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:48:49.031 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.035+0000 7f01b21c2700 1 -- 172.21.15.59:0/453457339 shutdown_connections 2024-01-13T11:48:49.031 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.035+0000 7f01b21c2700 1 -- 172.21.15.59:0/453457339 wait complete. 2024-01-13T11:48:49.031 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.036+0000 7f01b21c2700 1 Processor -- start 2024-01-13T11:48:49.031 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.036+0000 7f01b21c2700 1 -- start start 2024-01-13T11:48:49.031 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.036+0000 7f01b21c2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ac0fd380 0x7f01ac10a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:48:49.031 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.036+0000 7f01b21c2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac10a8a0 con 0x7f01ac0fd380 2024-01-13T11:48:49.031 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.036+0000 7f01ab7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ac0fd380 0x7f01ac10a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:48:49.031 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.036+0000 7f01ab7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ac0fd380 0x7f01ac10a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59590/0 (socket says 172.21.15.59:59590) 2024-01-13T11:48:49.031 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.036+0000 7f01ab7fe700 1 -- 172.21.15.59:0/2122569339 learned_addr learned my addr 172.21.15.59:0/2122569339 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:48:49.031 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.037+0000 7f01ab7fe700 1 -- 172.21.15.59:0/2122569339 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c00b200 con 0x7f01ac0fd380 2024-01-13T11:48:49.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.037+0000 7f01ab7fe700 1 --2- 172.21.15.59:0/2122569339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ac0fd380 0x7f01ac10a3b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f019c006c70 tx=0x7f019c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:48:49.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.037+0000 7f0193fff700 1 -- 172.21.15.59:0/2122569339 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f019c011b40 con 0x7f01ac0fd380 2024-01-13T11:48:49.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.037+0000 7f0193fff700 1 -- 172.21.15.59:0/2122569339 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c011ca0 con 0x7f01ac0fd380 2024-01-13T11:48:49.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.037+0000 7f0193fff700 1 -- 172.21.15.59:0/2122569339 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f019c01d430 con 0x7f01ac0fd380 2024-01-13T11:48:49.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.037+0000 7f01b21c2700 1 -- 172.21.15.59:0/2122569339 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ac10aaa0 con 0x7f01ac0fd380 2024-01-13T11:48:49.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.037+0000 7f01b21c2700 1 -- 172.21.15.59:0/2122569339 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ac10aeb0 con 0x7f01ac0fd380 2024-01-13T11:48:49.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.038+0000 7f0193fff700 1 -- 172.21.15.59:0/2122569339 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f019c00f070 con 0x7f01ac0fd380 2024-01-13T11:48:49.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.038+0000 7f0193fff700 1 -- 172.21.15.59:0/2122569339 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f019c027460 con 0x7f01ac0fd380 2024-01-13T11:48:49.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.038+0000 7f01b21c2700 1 -- 172.21.15.59:0/2122569339 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ac059100 con 0x7f01ac0fd380 2024-01-13T11:48:49.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.040+0000 7f0193fff700 1 -- 172.21.15.59:0/2122569339 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f019c016070 con 0x7f01ac0fd380 2024-01-13T11:48:49.097 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.107+0000 7f01b21c2700 1 -- 172.21.15.59:0/2122569339 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f01ac045ad0 con 0x7f01ac0fd380 2024-01-13T11:48:49.103 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.110+0000 7f0193fff700 1 -- 172.21.15.59:0/2122569339 <== mon.0 v2:172.21.15.59:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f019c01d620 con 0x7f01ac0fd380 2024-01-13T11:48:49.103 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.110+0000 7f0193fff700 1 -- 172.21.15.59:0/2122569339 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7f019c0308c0 con 0x7f01ac0fd380 2024-01-13T11:48:49.103 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.103 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: [global] 2024-01-13T11:48:49.103 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: fsid = a3276948-b209-11ee-95ac-87774f69a715 2024-01-13T11:48:49.103 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.59:3300,v1:172.21.15.59:6789] 2024-01-13T11:48:49.103 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-13T11:48:49.103 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-13T11:48:49.103 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-13T11:48:49.103 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-13T11:48:49.103 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.104 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: [mgr] 2024-01-13T11:48:49.104 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-13T11:48:49.104 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.104 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: [osd] 2024-01-13T11:48:49.104 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-01-13T11:48:49.104 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-01-13T11:48:49.104 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-01-13T11:48:49.104 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-13T11:48:49.104 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-01-13T11:48:49.104 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-13T11:48:49.104 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.111+0000 7f01b21c2700 1 -- 172.21.15.59:0/2122569339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ac0fd380 msgr2=0x7f01ac10a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:48:49.104 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.111+0000 7f01b21c2700 1 --2- 172.21.15.59:0/2122569339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ac0fd380 0x7f01ac10a3b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f019c006c70 tx=0x7f019c016000).stop 2024-01-13T11:48:49.104 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.111+0000 7f01b21c2700 1 -- 172.21.15.59:0/2122569339 shutdown_connections 2024-01-13T11:48:49.105 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.111+0000 7f01b21c2700 1 --2- 172.21.15.59:0/2122569339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ac0fd380 0x7f01ac10a3b0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:48:49.105 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.111+0000 7f01b21c2700 1 -- 172.21.15.59:0/2122569339 >> 172.21.15.59:0/2122569339 conn(0x7f01ac0f5b40 msgr2=0x7f01ac0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:48:49.105 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.111+0000 7f01b21c2700 1 -- 172.21.15.59:0/2122569339 shutdown_connections 2024-01-13T11:48:49.105 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.111+0000 7f01b21c2700 1 -- 172.21.15.59:0/2122569339 wait complete. 2024-01-13T11:48:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:Generating new minimal ceph.conf... 2024-01-13T11:48:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z -v /var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/mon.smithi059:/var/lib/ceph/mon/ceph-smithi059:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi059/config 2024-01-13T11:48:49.835 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.843+0000 7f57a157d700 1 Processor -- start 2024-01-13T11:48:49.835 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.843+0000 7f57a157d700 1 -- start start 2024-01-13T11:48:49.835 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.844+0000 7f57a157d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f579c0703f0 0x7f579c0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:48:49.836 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.844+0000 7f57a157d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579c0fee10 con 0x7f579c0703f0 2024-01-13T11:48:49.836 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.844+0000 7f579affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f579c0703f0 0x7f579c0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:48:49.836 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.844+0000 7f579affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f579c0703f0 0x7f579c0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59690/0 (socket says 172.21.15.59:59690) 2024-01-13T11:48:49.836 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.844+0000 7f579affd700 1 -- 172.21.15.59:0/348866724 learned_addr learned my addr 172.21.15.59:0/348866724 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:48:49.836 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.844+0000 7f579affd700 1 -- 172.21.15.59:0/348866724 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f579c0fef50 con 0x7f579c0703f0 2024-01-13T11:48:49.836 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.844+0000 7f579affd700 1 --2- 172.21.15.59:0/348866724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f579c0703f0 0x7f579c0fe920 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f578c00b5f0 tx=0x7f578c006ee0).ready entity=mon.0 client_cookie=9ce9c3a8fab47659 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:48:49.836 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.844+0000 7f5799ffb700 1 -- 172.21.15.59:0/348866724 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f578c00d600 con 0x7f579c0703f0 2024-01-13T11:48:49.836 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.845+0000 7f5799ffb700 1 -- 172.21.15.59:0/348866724 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f578c00dbd0 con 0x7f579c0703f0 2024-01-13T11:48:49.836 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.845+0000 7f5799ffb700 1 -- 172.21.15.59:0/348866724 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f578c011c00 con 0x7f579c0703f0 2024-01-13T11:48:49.836 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.845+0000 7f57a157d700 1 -- 172.21.15.59:0/348866724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f579c0703f0 msgr2=0x7f579c0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:48:49.836 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.845+0000 7f57a157d700 1 --2- 172.21.15.59:0/348866724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f579c0703f0 0x7f579c0fe920 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f578c00b5f0 tx=0x7f578c006ee0).stop 2024-01-13T11:48:49.836 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.845+0000 7f57a157d700 1 -- 172.21.15.59:0/348866724 shutdown_connections 2024-01-13T11:48:49.836 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.845+0000 7f57a157d700 1 --2- 172.21.15.59:0/348866724 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f579c0703f0 0x7f579c0fe920 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:48:49.837 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.845+0000 7f57a157d700 1 -- 172.21.15.59:0/348866724 >> 172.21.15.59:0/348866724 conn(0x7f579c0f63c0 msgr2=0x7f579c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:48:49.837 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.845+0000 7f57a157d700 1 -- 172.21.15.59:0/348866724 shutdown_connections 2024-01-13T11:48:49.837 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.845+0000 7f57a157d700 1 -- 172.21.15.59:0/348866724 wait complete. 2024-01-13T11:48:49.839 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.846+0000 7f57a157d700 1 Processor -- start 2024-01-13T11:48:49.839 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.846+0000 7f57a157d700 1 -- start start 2024-01-13T11:48:49.839 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.846+0000 7f57a157d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f579c0703f0 0x7f579c0fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:48:49.839 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.846+0000 7f57a157d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579c0fa820 con 0x7f579c0703f0 2024-01-13T11:48:49.839 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.847+0000 7f579affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f579c0703f0 0x7f579c0fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:48:49.839 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.847+0000 7f579affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f579c0703f0 0x7f579c0fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59692/0 (socket says 172.21.15.59:59692) 2024-01-13T11:48:49.839 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.847+0000 7f579affd700 1 -- 172.21.15.59:0/1220364597 learned_addr learned my addr 172.21.15.59:0/1220364597 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:48:49.839 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.847+0000 7f579affd700 1 -- 172.21.15.59:0/1220364597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578c00b200 con 0x7f579c0703f0 2024-01-13T11:48:49.839 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.847+0000 7f579affd700 1 --2- 172.21.15.59:0/1220364597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f579c0703f0 0x7f579c0fc3a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f578c00b190 tx=0x7f578c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:48:49.840 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.847+0000 7f57937fe700 1 -- 172.21.15.59:0/1220364597 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f578c01b410 con 0x7f579c0703f0 2024-01-13T11:48:49.840 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.847+0000 7f57937fe700 1 -- 172.21.15.59:0/1220364597 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f578c01b9e0 con 0x7f579c0703f0 2024-01-13T11:48:49.840 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.847+0000 7f57a157d700 1 -- 172.21.15.59:0/1220364597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f579c0fa9c0 con 0x7f579c0703f0 2024-01-13T11:48:49.840 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.847+0000 7f57937fe700 1 -- 172.21.15.59:0/1220364597 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f578c01a880 con 0x7f579c0703f0 2024-01-13T11:48:49.840 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.847+0000 7f57a157d700 1 -- 172.21.15.59:0/1220364597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f579c0fadd0 con 0x7f579c0703f0 2024-01-13T11:48:49.840 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.847+0000 7f57937fe700 1 -- 172.21.15.59:0/1220364597 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f578c00f070 con 0x7f579c0703f0 2024-01-13T11:48:49.840 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.847+0000 7f57937fe700 1 -- 172.21.15.59:0/1220364597 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f578c025b70 con 0x7f579c0703f0 2024-01-13T11:48:49.840 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.848+0000 7f57a157d700 1 -- 172.21.15.59:0/1220364597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f579c045ad0 con 0x7f579c0703f0 2024-01-13T11:48:49.840 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.851+0000 7f57937fe700 1 -- 172.21.15.59:0/1220364597 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f578c016070 con 0x7f579c0703f0 2024-01-13T11:48:49.889 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:49 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38356]: audit 2024-01-13T11:48:49.108120+0000 mon.smithi059 (mon.0) 8 : audit [INF] from='client.? 172.21.15.59:0/2122569339' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-13T11:48:49.889 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:49 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38356]: audit 2024-01-13T11:48:49.111198+0000 mon.smithi059 (mon.0) 9 : audit [INF] from='client.? 172.21.15.59:0/2122569339' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-13T11:48:49.908 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.918+0000 7f57a157d700 1 -- 172.21.15.59:0/1220364597 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f579c0008d0 con 0x7f579c0703f0 2024-01-13T11:48:49.908 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.918+0000 7f57937fe700 1 -- 172.21.15.59:0/1220364597 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f578c025db0 con 0x7f579c0703f0 2024-01-13T11:48:49.909 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.919+0000 7f57a157d700 1 -- 172.21.15.59:0/1220364597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f579c0703f0 msgr2=0x7f579c0fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:48:49.909 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.919+0000 7f57a157d700 1 --2- 172.21.15.59:0/1220364597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f579c0703f0 0x7f579c0fc3a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f578c00b190 tx=0x7f578c006ee0).stop 2024-01-13T11:48:49.909 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.919+0000 7f57a157d700 1 -- 172.21.15.59:0/1220364597 shutdown_connections 2024-01-13T11:48:49.909 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.919+0000 7f57a157d700 1 --2- 172.21.15.59:0/1220364597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f579c0703f0 0x7f579c0fc3a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:48:49.909 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.919+0000 7f57a157d700 1 -- 172.21.15.59:0/1220364597 >> 172.21.15.59:0/1220364597 conn(0x7f579c0f63c0 msgr2=0x7f579c0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:48:49.909 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.919+0000 7f57a157d700 1 -- 172.21.15.59:0/1220364597 shutdown_connections 2024-01-13T11:48:49.909 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:49.919+0000 7f57a157d700 1 -- 172.21.15.59:0/1220364597 wait complete. 2024-01-13T11:48:50.355 INFO:teuthology.orchestra.run.smithi059.stderr:Restarting the monitor... 2024-01-13T11:48:50.355 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl restart ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi059 2024-01-13T11:48:50.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:50 smithi059 systemd[1]: Stopping Ceph mon.smithi059 for a3276948-b209-11ee-95ac-87774f69a715... 2024-01-13T11:48:50.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:50 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38356]: debug 2024-01-13T11:48:50.438+0000 7febf6ce1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi059 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-13T11:48:50.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:50 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38356]: debug 2024-01-13T11:48:50.438+0000 7febf6ce1700 -1 mon.smithi059@0(leader) e1 *** Got Signal Terminated *** 2024-01-13T11:48:51.085 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:50 smithi059 podman[38675]: ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059 2024-01-13T11:48:51.085 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:50 smithi059 systemd[1]: ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi059.service: Succeeded. 2024-01-13T11:48:51.085 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:50 smithi059 systemd[1]: Stopped Ceph mon.smithi059 for a3276948-b209-11ee-95ac-87774f69a715. 2024-01-13T11:48:51.085 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:50 smithi059 systemd[1]: Starting Ceph mon.smithi059 for a3276948-b209-11ee-95ac-87774f69a715... 2024-01-13T11:48:51.156 INFO:teuthology.orchestra.run.smithi059.stderr:Setting mon public_network to 172.21.0.0/20 2024-01-13T11:48:51.156 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-01-13T11:48:51.544 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.553+0000 7fb0196ce700 1 Processor -- start 2024-01-13T11:48:51.546 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.553+0000 7fb0196ce700 1 -- start start 2024-01-13T11:48:51.546 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.554+0000 7fb0196ce700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0140703d0 0x7fb014100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:48:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.554+0000 7fb0196ce700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0140fc730 con 0x7fb0140703d0 2024-01-13T11:48:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.554+0000 7fb012ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0140703d0 0x7fb014100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:48:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.554+0000 7fb012ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0140703d0 0x7fb014100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59696/0 (socket says 172.21.15.59:59696) 2024-01-13T11:48:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.554+0000 7fb012ffd700 1 -- 172.21.15.59:0/3157386919 learned_addr learned my addr 172.21.15.59:0/3157386919 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:48:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.554+0000 7fb012ffd700 1 -- 172.21.15.59:0/3157386919 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb014101230 con 0x7fb0140703d0 2024-01-13T11:48:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.554+0000 7fb012ffd700 1 --2- 172.21.15.59:0/3157386919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0140703d0 0x7fb014100d40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb00400b5f0 tx=0x7fb004006ee0).ready entity=mon.0 client_cookie=6100663ebe71354f server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:48:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.555+0000 7fb011ffb700 1 -- 172.21.15.59:0/3157386919 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb00400d600 con 0x7fb0140703d0 2024-01-13T11:48:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.555+0000 7fb011ffb700 1 -- 172.21.15.59:0/3157386919 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb00400dbd0 con 0x7fb0140703d0 2024-01-13T11:48:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.555+0000 7fb011ffb700 1 -- 172.21.15.59:0/3157386919 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb004011c00 con 0x7fb0140703d0 2024-01-13T11:48:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.555+0000 7fb0196ce700 1 -- 172.21.15.59:0/3157386919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0140703d0 msgr2=0x7fb014100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:48:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.555+0000 7fb0196ce700 1 --2- 172.21.15.59:0/3157386919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0140703d0 0x7fb014100d40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb00400b5f0 tx=0x7fb004006ee0).stop 2024-01-13T11:48:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.555+0000 7fb0196ce700 1 -- 172.21.15.59:0/3157386919 shutdown_connections 2024-01-13T11:48:51.547 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.555+0000 7fb0196ce700 1 --2- 172.21.15.59:0/3157386919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0140703d0 0x7fb014100d40 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:48:51.548 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.555+0000 7fb0196ce700 1 -- 172.21.15.59:0/3157386919 >> 172.21.15.59:0/3157386919 conn(0x7fb0140f63c0 msgr2=0x7fb0140f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:48:51.548 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.555+0000 7fb0196ce700 1 -- 172.21.15.59:0/3157386919 shutdown_connections 2024-01-13T11:48:51.548 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.555+0000 7fb0196ce700 1 -- 172.21.15.59:0/3157386919 wait complete. 2024-01-13T11:48:51.548 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.556+0000 7fb0196ce700 1 Processor -- start 2024-01-13T11:48:51.548 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.556+0000 7fb0196ce700 1 -- start start 2024-01-13T11:48:51.548 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.556+0000 7fb0196ce700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0140703d0 0x7fb01410cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:48:51.548 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.556+0000 7fb0196ce700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01410d100 con 0x7fb0140703d0 2024-01-13T11:48:51.548 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.556+0000 7fb012ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0140703d0 0x7fb01410cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:48:51.550 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.556+0000 7fb012ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0140703d0 0x7fb01410cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59698/0 (socket says 172.21.15.59:59698) 2024-01-13T11:48:51.550 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.556+0000 7fb012ffd700 1 -- 172.21.15.59:0/291398772 learned_addr learned my addr 172.21.15.59:0/291398772 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:48:51.550 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.557+0000 7fb012ffd700 1 -- 172.21.15.59:0/291398772 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00400b200 con 0x7fb0140703d0 2024-01-13T11:48:51.550 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.557+0000 7fb012ffd700 1 --2- 172.21.15.59:0/291398772 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0140703d0 0x7fb01410cc10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fb00400e040 tx=0x7fb004006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:48:51.550 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.557+0000 7fb00b7fe700 1 -- 172.21.15.59:0/291398772 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb00400d9f0 con 0x7fb0140703d0 2024-01-13T11:48:51.550 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.557+0000 7fb00b7fe700 1 -- 172.21.15.59:0/291398772 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb00401a9f0 con 0x7fb0140703d0 2024-01-13T11:48:51.550 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.557+0000 7fb0196ce700 1 -- 172.21.15.59:0/291398772 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb01410d300 con 0x7fb0140703d0 2024-01-13T11:48:51.551 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.557+0000 7fb00b7fe700 1 -- 172.21.15.59:0/291398772 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb00401b8c0 con 0x7fb0140703d0 2024-01-13T11:48:51.551 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.557+0000 7fb0196ce700 1 -- 172.21.15.59:0/291398772 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb01410d710 con 0x7fb0140703d0 2024-01-13T11:48:51.551 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.557+0000 7fb00b7fe700 1 -- 172.21.15.59:0/291398772 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fb00400f070 con 0x7fb0140703d0 2024-01-13T11:48:51.551 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.557+0000 7fb00b7fe700 1 -- 172.21.15.59:0/291398772 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb004025ae0 con 0x7fb0140703d0 2024-01-13T11:48:51.551 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.558+0000 7fb0196ce700 1 -- 172.21.15.59:0/291398772 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb014045ad0 con 0x7fb0140703d0 2024-01-13T11:48:51.552 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.561+0000 7fb00b7fe700 1 -- 172.21.15.59:0/291398772 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fb004016070 con 0x7fb0140703d0 2024-01-13T11:48:51.616 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.626+0000 7fb0196ce700 1 -- 172.21.15.59:0/291398772 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fb014101ee0 con 0x7fb0140703d0 2024-01-13T11:48:51.621 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.631+0000 7fb00b7fe700 1 -- 172.21.15.59:0/291398772 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fb00401bdc0 con 0x7fb0140703d0 2024-01-13T11:48:51.623 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.632+0000 7fb0196ce700 1 -- 172.21.15.59:0/291398772 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0140703d0 msgr2=0x7fb01410cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:48:51.623 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.632+0000 7fb0196ce700 1 --2- 172.21.15.59:0/291398772 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0140703d0 0x7fb01410cc10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fb00400e040 tx=0x7fb004006ee0).stop 2024-01-13T11:48:51.623 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.632+0000 7fb0196ce700 1 -- 172.21.15.59:0/291398772 shutdown_connections 2024-01-13T11:48:51.623 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.632+0000 7fb0196ce700 1 --2- 172.21.15.59:0/291398772 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0140703d0 0x7fb01410cc10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:48:51.623 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.632+0000 7fb0196ce700 1 -- 172.21.15.59:0/291398772 >> 172.21.15.59:0/291398772 conn(0x7fb0140f63c0 msgr2=0x7fb0140f7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:48:51.623 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.632+0000 7fb0196ce700 1 -- 172.21.15.59:0/291398772 shutdown_connections 2024-01-13T11:48:51.624 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:51.632+0000 7fb0196ce700 1 -- 172.21.15.59:0/291398772 wait complete. 2024-01-13T11:48:51.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.158+0000 7f0a067c3700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-13T11:48:51.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.158+0000 7f0a067c3700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-01-13T11:48:51.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.158+0000 7f0a067c3700 0 pidfile_write: ignore empty --pid-file 2024-01-13T11:48:51.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.162+0000 7f0a067c3700 0 load: jerasure load: lrc load: isa 2024-01-13T11:48:51.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 bash[38746]: d08fac799508eeae7e3615952d9e2c135c47bcdc3db5b19d2fbca58f26900a06 2024-01-13T11:48:51.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: RocksDB version: 6.8.1 2024-01-13T11:48:51.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-13T11:48:51.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Compile date May 13 2021 2024-01-13T11:48:51.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: DB SUMMARY 2024-01-13T11:48:51.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.641 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: CURRENT file: CURRENT 2024-01-13T11:48:51.641 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.641 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: IDENTITY file: IDENTITY 2024-01-13T11:48:51.641 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.641 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-01-13T11:48:51.641 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.641 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi059/store.db dir, Total Num: 1, files: 000004.sst 2024-01-13T11:48:51.641 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.641 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi059/store.db: 000006.log size: 92918 ; 2024-01-13T11:48:51.641 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.641 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.error_if_exists: 0 2024-01-13T11:48:51.641 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.create_if_missing: 0 2024-01-13T11:48:51.641 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.paranoid_checks: 1 2024-01-13T11:48:51.641 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.env: 0x56234a8d81c0 2024-01-13T11:48:51.642 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.fs: Posix File System 2024-01-13T11:48:51.642 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.info_log: 0x56234cf39960 2024-01-13T11:48:51.642 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-13T11:48:51.642 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.statistics: (nil) 2024-01-13T11:48:51.642 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.use_fsync: 0 2024-01-13T11:48:51.642 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.max_log_file_size: 0 2024-01-13T11:48:51.642 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-13T11:48:51.642 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-13T11:48:51.642 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-13T11:48:51.642 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-13T11:48:51.642 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.allow_fallocate: 1 2024-01-13T11:48:51.643 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-13T11:48:51.643 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-13T11:48:51.643 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.use_direct_reads: 0 2024-01-13T11:48:51.643 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-13T11:48:51.643 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-13T11:48:51.643 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.db_log_dir: 2024-01-13T11:48:51.643 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi059/store.db 2024-01-13T11:48:51.643 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-13T11:48:51.643 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.max_subcompactions: 1 2024-01-13T11:48:51.643 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.max_background_flushes: -1 2024-01-13T11:48:51.643 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-13T11:48:51.643 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-13T11:48:51.644 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-13T11:48:51.644 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-13T11:48:51.644 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-13T11:48:51.644 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-13T11:48:51.644 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-13T11:48:51.644 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.write_buffer_manager: 0x56234dbae030 2024-01-13T11:48:51.644 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-13T11:48:51.644 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-13T11:48:51.644 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-13T11:48:51.644 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-13T11:48:51.645 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-13T11:48:51.645 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-13T11:48:51.645 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-13T11:48:51.645 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-13T11:48:51.645 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-13T11:48:51.645 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.unordered_write: 0 2024-01-13T11:48:51.645 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-13T11:48:51.646 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-13T11:48:51.647 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-13T11:48:51.647 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-13T11:48:51.647 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.row_cache: None 2024-01-13T11:48:51.647 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.wal_filter: None 2024-01-13T11:48:51.647 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-13T11:48:51.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-13T11:48:51.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.preserve_deletes: 0 2024-01-13T11:48:51.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.two_write_queues: 0 2024-01-13T11:48:51.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-13T11:48:51.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.atomic_flush: 0 2024-01-13T11:48:51.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-13T11:48:51.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-13T11:48:51.648 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-13T11:48:51.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.log_readahead_size: 0 2024-01-13T11:48:51.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-13T11:48:51.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.max_background_jobs: 2 2024-01-13T11:48:51.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.max_background_compactions: -1 2024-01-13T11:48:51.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-13T11:48:51.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-13T11:48:51.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-13T11:48:51.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-13T11:48:51.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-13T11:48:51.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-13T11:48:51.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-13T11:48:51.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-13T11:48:51.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.max_open_files: -1 2024-01-13T11:48:51.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-13T11:48:51.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-13T11:48:51.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-13T11:48:51.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-13T11:48:51.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Compression algorithms supported: 2024-01-13T11:48:51.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-13T11:48:51.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: kZSTD supported: 0 2024-01-13T11:48:51.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: kXpressCompression supported: 0 2024-01-13T11:48:51.651 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-13T11:48:51.651 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: kLZ4Compression supported: 1 2024-01-13T11:48:51.651 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: kBZip2Compression supported: 0 2024-01-13T11:48:51.651 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: kZlibCompression supported: 1 2024-01-13T11:48:51.651 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: kSnappyCompression supported: 1 2024-01-13T11:48:51.651 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-13T11:48:51.654 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi059/store.db/MANIFEST-000005 2024-01-13T11:48:51.655 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.655 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-13T11:48:51.655 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.655 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-13T11:48:51.655 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.merge_operator: 2024-01-13T11:48:51.656 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.compaction_filter: None 2024-01-13T11:48:51.656 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-13T11:48:51.656 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-13T11:48:51.656 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.163+0000 7f0a067c3700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-13T11:48:51.656 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56234ce50128) 2024-01-13T11:48:51.656 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cache_index_and_filter_blocks: 1 2024-01-13T11:48:51.657 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-13T11:48:51.657 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-13T11:48:51.657 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: pin_top_level_index_and_filter: 1 2024-01-13T11:48:51.657 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: index_type: 0 2024-01-13T11:48:51.657 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: data_block_index_type: 0 2024-01-13T11:48:51.657 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: index_shortening: 1 2024-01-13T11:48:51.657 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: data_block_hash_table_util_ratio: 0.750000 2024-01-13T11:48:51.657 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: hash_index_allow_collision: 1 2024-01-13T11:48:51.658 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: checksum: 1 2024-01-13T11:48:51.658 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: no_block_cache: 0 2024-01-13T11:48:51.658 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: block_cache: 0x56234ce86f10 2024-01-13T11:48:51.658 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: block_cache_name: BinnedLRUCache 2024-01-13T11:48:51.658 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: block_cache_options: 2024-01-13T11:48:51.658 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: capacity : 536870912 2024-01-13T11:48:51.658 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: num_shard_bits : 4 2024-01-13T11:48:51.658 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: strict_capacity_limit : 0 2024-01-13T11:48:51.658 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: high_pri_pool_ratio: 0.000 2024-01-13T11:48:51.659 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: block_cache_compressed: (nil) 2024-01-13T11:48:51.659 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: persistent_cache: (nil) 2024-01-13T11:48:51.659 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: block_size: 4096 2024-01-13T11:48:51.659 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: block_size_deviation: 10 2024-01-13T11:48:51.659 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: block_restart_interval: 16 2024-01-13T11:48:51.659 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: index_block_restart_interval: 1 2024-01-13T11:48:51.659 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: metadata_block_size: 4096 2024-01-13T11:48:51.659 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: partition_filters: 0 2024-01-13T11:48:51.660 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: use_delta_encoding: 1 2024-01-13T11:48:51.660 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-13T11:48:51.660 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: whole_key_filtering: 1 2024-01-13T11:48:51.660 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: verify_compression: 0 2024-01-13T11:48:51.660 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: read_amp_bytes_per_bit: 0 2024-01-13T11:48:51.660 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: format_version: 2 2024-01-13T11:48:51.660 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: enable_index_compression: 1 2024-01-13T11:48:51.661 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: block_align: 0 2024-01-13T11:48:51.661 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.661 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-13T11:48:51.661 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-13T11:48:51.662 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.compression: NoCompression 2024-01-13T11:48:51.662 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-13T11:48:51.662 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-13T11:48:51.662 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-13T11:48:51.662 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.num_levels: 7 2024-01-13T11:48:51.662 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-13T11:48:51.663 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-13T11:48:51.663 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-13T11:48:51.663 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-13T11:48:51.663 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-13T11:48:51.663 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-13T11:48:51.664 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-13T11:48:51.664 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-13T11:48:51.664 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-13T11:48:51.664 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-13T11:48:51.664 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-13T11:48:51.664 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-13T11:48:51.665 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-13T11:48:51.665 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-13T11:48:51.665 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-13T11:48:51.665 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-13T11:48:51.666 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-13T11:48:51.666 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-13T11:48:51.666 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-13T11:48:51.666 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-13T11:48:51.666 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-13T11:48:51.666 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-13T11:48:51.667 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-13T11:48:51.667 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-13T11:48:51.667 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-13T11:48:51.667 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-13T11:48:51.667 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-13T11:48:51.667 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-13T11:48:51.668 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-13T11:48:51.668 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-13T11:48:51.668 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-13T11:48:51.668 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-13T11:48:51.669 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-13T11:48:51.669 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-13T11:48:51.669 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-13T11:48:51.669 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-13T11:48:51.669 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-13T11:48:51.669 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-13T11:48:51.670 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-13T11:48:51.670 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-13T11:48:51.670 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-13T11:48:51.670 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-13T11:48:51.670 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-13T11:48:51.670 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-13T11:48:51.671 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-13T11:48:51.671 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-13T11:48:51.671 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-13T11:48:51.671 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.table_properties_collectors: 2024-01-13T11:48:51.671 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.inplace_update_support: 0 2024-01-13T11:48:51.671 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-13T11:48:51.672 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-13T11:48:51.672 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-13T11:48:51.672 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-13T11:48:51.672 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.bloom_locality: 0 2024-01-13T11:48:51.672 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.max_successive_merges: 0 2024-01-13T11:48:51.673 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-13T11:48:51.673 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-13T11:48:51.673 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-13T11:48:51.674 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-13T11:48:51.674 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.ttl: 2592000 2024-01-13T11:48:51.674 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-13T11:48:51.674 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi059/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-13T11:48:51.674 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.674 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-01-13T11:48:51.675 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.675 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705146531165381, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-01-13T11:48:51.675 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.164+0000 7f0a067c3700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-01-13T11:48:51.675 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 systemd[1]: Started Ceph mon.smithi059 for a3276948-b209-11ee-95ac-87774f69a715. 2024-01-13T11:48:51.675 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.165+0000 7f0a067c3700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-13T11:48:51.675 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.166+0000 7f0a067c3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705146531166640, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88698, "table_properties": {"data_size": 86813, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79954, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705146531, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-13T11:48:51.676 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.166+0000 7f0a067c3700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-01-13T11:48:51.676 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.676 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.166+0000 7f0a067c3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705146531167077, "job": 1, "event": "recovery_finished"} 2024-01-13T11:48:51.676 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.167+0000 7f0a067c3700 4 rocksdb: DB pointer 0x56234cf4b800 2024-01-13T11:48:51.676 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.167+0000 7f09ed5db700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-13T11:48:51.676 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.167+0000 7f09ed5db700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-13T11:48:51.677 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: ** DB Stats ** 2024-01-13T11:48:51.677 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T11:48:51.677 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-13T11:48:51.677 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-13T11:48:51.677 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T11:48:51.678 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-13T11:48:51.678 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-13T11:48:51.678 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T11:48:51.678 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.678 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: ** Compaction Stats [default] ** 2024-01-13T11:48:51.678 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:48:51.679 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:48:51.679 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 106.3 0.00 0.00 1 0.001 0 0 2024-01-13T11:48:51.679 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 106.3 0.00 0.00 1 0.001 0 0 2024-01-13T11:48:51.679 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 106.3 0.00 0.00 1 0.001 0 0 2024-01-13T11:48:51.679 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.679 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: ** Compaction Stats [default] ** 2024-01-13T11:48:51.680 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:48:51.680 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:48:51.680 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 106.3 0.00 0.00 1 0.001 0 0 2024-01-13T11:48:51.680 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T11:48:51.680 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T11:48:51.681 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T11:48:51.681 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T11:48:51.681 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T11:48:51.681 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: AddFile(Keys): cumulative 0, interval 0 2024-01-13T11:48:51.681 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Cumulative compaction: 0.00 GB write, 23.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:48:51.681 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Interval compaction: 0.00 GB write, 23.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:48:51.682 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T11:48:51.682 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.682 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: ** File Read Latency Histogram By Level [default] ** 2024-01-13T11:48:51.682 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.682 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: ** Compaction Stats [default] ** 2024-01-13T11:48:51.682 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:48:51.683 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:48:51.683 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 106.3 0.00 0.00 1 0.001 0 0 2024-01-13T11:48:51.683 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 106.3 0.00 0.00 1 0.001 0 0 2024-01-13T11:48:51.683 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T11:48:51.683 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.683 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: ** Compaction Stats [default] ** 2024-01-13T11:48:51.684 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:48:51.684 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:48:51.684 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 106.3 0.00 0.00 1 0.001 0 0 2024-01-13T11:48:51.684 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T11:48:51.684 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T11:48:51.684 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T11:48:51.685 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T11:48:51.685 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T11:48:51.685 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: AddFile(Keys): cumulative 0, interval 0 2024-01-13T11:48:51.685 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Cumulative compaction: 0.00 GB write, 23.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:48:51.685 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:48:51.686 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T11:48:51.686 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.686 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: ** File Read Latency Histogram By Level [default] ** 2024-01-13T11:48:51.686 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.686 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.168+0000 7f0a067c3700 0 starting mon.smithi059 rank 0 at public addrs [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] at bind addrs [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi059 fsid a3276948-b209-11ee-95ac-87774f69a715 2024-01-13T11:48:51.686 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.169+0000 7f0a067c3700 1 mon.smithi059@-1(???) e1 preinit fsid a3276948-b209-11ee-95ac-87774f69a715 2024-01-13T11:48:51.687 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.169+0000 7f0a067c3700 0 mon.smithi059@-1(???).mds e1 new map 2024-01-13T11:48:51.687 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.169+0000 7f0a067c3700 0 mon.smithi059@-1(???).mds e1 print_map 2024-01-13T11:48:51.687 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: e1 2024-01-13T11:48:51.687 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-13T11:48:51.687 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-13T11:48:51.687 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: legacy client fscid: -1 2024-01-13T11:48:51.688 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.688 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: No filesystems configured 2024-01-13T11:48:51.688 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: 2024-01-13T11:48:51.688 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.169+0000 7f0a067c3700 0 mon.smithi059@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-13T11:48:51.688 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.169+0000 7f0a067c3700 0 mon.smithi059@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T11:48:51.688 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.169+0000 7f0a067c3700 0 mon.smithi059@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T11:48:51.689 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.169+0000 7f0a067c3700 0 mon.smithi059@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T11:48:51.689 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:48:49.108120+0000 mon.smithi059 (mon.0) 8 : audit [INF] from='client.? 172.21.15.59:0/2122569339' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-13T11:48:51.689 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:48:49.111198+0000 mon.smithi059 (mon.0) 9 : audit [INF] from='client.? 172.21.15.59:0/2122569339' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-13T11:48:51.689 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.169+0000 7f0a067c3700 1 mon.smithi059@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-13T11:48:51.689 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.170+0000 7f0a067c3700 20 expand_channel_meta expand map: {default=false} 2024-01-13T11:48:51.690 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.170+0000 7f0a067c3700 20 expand_channel_meta from 'false' to 'false' 2024-01-13T11:48:51.690 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.170+0000 7f0a067c3700 20 expand_channel_meta expanded map: {default=false} 2024-01-13T11:48:51.690 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.170+0000 7f0a067c3700 20 expand_channel_meta expand map: {default=info} 2024-01-13T11:48:51.690 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.170+0000 7f0a067c3700 20 expand_channel_meta from 'info' to 'info' 2024-01-13T11:48:51.690 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.170+0000 7f0a067c3700 20 expand_channel_meta expanded map: {default=info} 2024-01-13T11:48:51.690 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.170+0000 7f0a067c3700 20 expand_channel_meta expand map: {default=daemon} 2024-01-13T11:48:51.690 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.170+0000 7f0a067c3700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-01-13T11:48:51.691 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.170+0000 7f0a067c3700 20 expand_channel_meta expanded map: {default=daemon} 2024-01-13T11:48:51.691 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.170+0000 7f0a067c3700 20 expand_channel_meta expand map: {default=debug} 2024-01-13T11:48:51.691 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.170+0000 7f0a067c3700 20 expand_channel_meta from 'debug' to 'debug' 2024-01-13T11:48:51.691 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:48:51.170+0000 7f0a067c3700 20 expand_channel_meta expanded map: {default=debug} 2024-01-13T11:48:51.691 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:48:51.185174+0000 mon.smithi059 (mon.0) 1 : cluster [INF] mon.smithi059 is new leader, mons smithi059 in quorum (ranks 0) 2024-01-13T11:48:51.691 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:48:51.185277+0000 mon.smithi059 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi059=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} 2024-01-13T11:48:51.691 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:48:51.185340+0000 mon.smithi059 (mon.0) 3 : cluster [DBG] fsmap 2024-01-13T11:48:51.691 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:48:51.185391+0000 mon.smithi059 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-13T11:48:51.692 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:48:51.185868+0000 mon.smithi059 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-01-13T11:48:52.030 INFO:teuthology.orchestra.run.smithi059.stderr:Wrote config to /etc/ceph/ceph.conf 2024-01-13T11:48:52.030 INFO:teuthology.orchestra.run.smithi059.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-13T11:48:52.031 INFO:teuthology.orchestra.run.smithi059.stderr:Creating mgr... 2024-01-13T11:48:52.031 INFO:teuthology.orchestra.run.smithi059.stderr:Verifying port 9283 ... 2024-01-13T11:48:52.033 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl daemon-reload 2024-01-13T11:48:52.133 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl stop ceph-a3276948-b209-11ee-95ac-87774f69a715@mgr.smithi059.hmyldj 2024-01-13T11:48:52.166 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl reset-failed ceph-a3276948-b209-11ee-95ac-87774f69a715@mgr.smithi059.hmyldj 2024-01-13T11:48:52.171 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Failed to reset failed state of unit ceph-a3276948-b209-11ee-95ac-87774f69a715@mgr.smithi059.hmyldj.service: Unit ceph-a3276948-b209-11ee-95ac-87774f69a715@mgr.smithi059.hmyldj.service not loaded. 2024-01-13T11:48:52.172 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl enable ceph-a3276948-b209-11ee-95ac-87774f69a715@mgr.smithi059.hmyldj 2024-01-13T11:48:52.178 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Created symlink /etc/systemd/system/ceph-a3276948-b209-11ee-95ac-87774f69a715.target.wants/ceph-a3276948-b209-11ee-95ac-87774f69a715@mgr.smithi059.hmyldj.service → /etc/systemd/system/ceph-a3276948-b209-11ee-95ac-87774f69a715@.service. 2024-01-13T11:48:52.266 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl start ceph-a3276948-b209-11ee-95ac-87774f69a715@mgr.smithi059.hmyldj 2024-01-13T11:48:52.625 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-13T11:48:52.633 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: disabled 2024-01-13T11:48:52.634 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl is-active firewalld.service 2024-01-13T11:48:52.643 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: inactive 2024-01-13T11:48:52.643 INFO:teuthology.orchestra.run.smithi059.stderr:firewalld.service is not enabled 2024-01-13T11:48:52.643 INFO:teuthology.orchestra.run.smithi059.stderr:Not possible to enable service . firewalld.service is not available 2024-01-13T11:48:52.643 INFO:teuthology.orchestra.run.smithi059.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-13T11:48:52.643 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-13T11:48:52.651 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: disabled 2024-01-13T11:48:52.651 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl is-active firewalld.service 2024-01-13T11:48:52.659 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: inactive 2024-01-13T11:48:52.660 INFO:teuthology.orchestra.run.smithi059.stderr:firewalld.service is not enabled 2024-01-13T11:48:52.660 INFO:teuthology.orchestra.run.smithi059.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-13T11:48:52.660 INFO:teuthology.orchestra.run.smithi059.stderr:Waiting for mgr to start... 2024-01-13T11:48:52.660 INFO:teuthology.orchestra.run.smithi059.stderr:Waiting for mgr... 2024-01-13T11:48:52.660 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-13T11:48:52.889 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:48:51.632171+0000 mon.smithi059 (mon.0) 6 : audit [INF] from='client.? 172.21.15.59:0/291398772' entity='client.admin' 2024-01-13T11:48:53.102 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.111+0000 7f10861ba700 1 Processor -- start 2024-01-13T11:48:53.102 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.111+0000 7f10861ba700 1 -- start start 2024-01-13T11:48:53.102 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.111+0000 7f10861ba700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10800703d0 0x7f1080100b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:48:53.102 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.111+0000 7f10861ba700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10800fc580 con 0x7f10800703d0 2024-01-13T11:48:53.102 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.112+0000 7f10851b8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10800703d0 0x7f1080100b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:48:53.102 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.112+0000 7f10851b8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10800703d0 0x7f1080100b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59708/0 (socket says 172.21.15.59:59708) 2024-01-13T11:48:53.102 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.112+0000 7f10851b8700 1 -- 172.21.15.59:0/3837965170 learned_addr learned my addr 172.21.15.59:0/3837965170 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:48:53.102 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.112+0000 7f10851b8700 1 -- 172.21.15.59:0/3837965170 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1080101080 con 0x7f10800703d0 2024-01-13T11:48:53.102 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.112+0000 7f10851b8700 1 --2- 172.21.15.59:0/3837965170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10800703d0 0x7f1080100b90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f107c00b5f0 tx=0x7f107c006ee0).ready entity=mon.0 client_cookie=80b535eeeefcf201 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:48:53.105 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.112+0000 7f106ffff700 1 -- 172.21.15.59:0/3837965170 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f107c00d600 con 0x7f10800703d0 2024-01-13T11:48:53.105 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.113+0000 7f106ffff700 1 -- 172.21.15.59:0/3837965170 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f107c00dbd0 con 0x7f10800703d0 2024-01-13T11:48:53.105 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.113+0000 7f106ffff700 1 -- 172.21.15.59:0/3837965170 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f107c011c00 con 0x7f10800703d0 2024-01-13T11:48:53.105 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.113+0000 7f10861ba700 1 -- 172.21.15.59:0/3837965170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10800703d0 msgr2=0x7f1080100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:48:53.105 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.113+0000 7f10861ba700 1 --2- 172.21.15.59:0/3837965170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10800703d0 0x7f1080100b90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f107c00b5f0 tx=0x7f107c006ee0).stop 2024-01-13T11:48:53.105 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.113+0000 7f10861ba700 1 -- 172.21.15.59:0/3837965170 shutdown_connections 2024-01-13T11:48:53.105 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.113+0000 7f10861ba700 1 --2- 172.21.15.59:0/3837965170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10800703d0 0x7f1080100b90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:48:53.105 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.113+0000 7f10861ba700 1 -- 172.21.15.59:0/3837965170 >> 172.21.15.59:0/3837965170 conn(0x7f10800f6210 msgr2=0x7f10800f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:48:53.105 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.113+0000 7f10861ba700 1 -- 172.21.15.59:0/3837965170 shutdown_connections 2024-01-13T11:48:53.105 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.113+0000 7f10861ba700 1 -- 172.21.15.59:0/3837965170 wait complete. 2024-01-13T11:48:53.105 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.114+0000 7f10861ba700 1 Processor -- start 2024-01-13T11:48:53.105 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.114+0000 7f10861ba700 1 -- start start 2024-01-13T11:48:53.106 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.114+0000 7f10861ba700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10800703d0 0x7f108010ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:48:53.106 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.114+0000 7f10861ba700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108010cf70 con 0x7f10800703d0 2024-01-13T11:48:53.106 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.114+0000 7f10851b8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10800703d0 0x7f108010ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:48:53.106 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.114+0000 7f10851b8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10800703d0 0x7f108010ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59716/0 (socket says 172.21.15.59:59716) 2024-01-13T11:48:53.106 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.114+0000 7f10851b8700 1 -- 172.21.15.59:0/1206731668 learned_addr learned my addr 172.21.15.59:0/1206731668 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:48:53.106 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.115+0000 7f10851b8700 1 -- 172.21.15.59:0/1206731668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c00b200 con 0x7f10800703d0 2024-01-13T11:48:53.106 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.115+0000 7f10851b8700 1 --2- 172.21.15.59:0/1206731668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10800703d0 0x7f108010ca80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f107c00e040 tx=0x7f107c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:48:53.106 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.115+0000 7f106dffb700 1 -- 172.21.15.59:0/1206731668 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f107c00d9f0 con 0x7f10800703d0 2024-01-13T11:48:53.106 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.115+0000 7f106dffb700 1 -- 172.21.15.59:0/1206731668 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f107c01a9f0 con 0x7f10800703d0 2024-01-13T11:48:53.107 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.115+0000 7f10861ba700 1 -- 172.21.15.59:0/1206731668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f108010d170 con 0x7f10800703d0 2024-01-13T11:48:53.107 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.115+0000 7f106dffb700 1 -- 172.21.15.59:0/1206731668 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f107c01b8c0 con 0x7f10800703d0 2024-01-13T11:48:53.107 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.115+0000 7f10861ba700 1 -- 172.21.15.59:0/1206731668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f108010d580 con 0x7f10800703d0 2024-01-13T11:48:53.107 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.115+0000 7f106dffb700 1 -- 172.21.15.59:0/1206731668 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f107c00f070 con 0x7f10800703d0 2024-01-13T11:48:53.107 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.115+0000 7f106dffb700 1 -- 172.21.15.59:0/1206731668 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f107c025ae0 con 0x7f10800703d0 2024-01-13T11:48:53.107 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.117+0000 7f10861ba700 1 -- 172.21.15.59:0/1206731668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1080045ad0 con 0x7f10800703d0 2024-01-13T11:48:53.108 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.119+0000 7f106dffb700 1 -- 172.21.15.59:0/1206731668 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f107c016070 con 0x7f10800703d0 2024-01-13T11:48:53.180 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.189+0000 7f10861ba700 1 -- 172.21.15.59:0/1206731668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1080101d30 con 0x7f10800703d0 2024-01-13T11:48:53.180 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.189+0000 7f106dffb700 1 -- 172.21.15.59:0/1206731668 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f107c01bdc0 con 0x7f10800703d0 2024-01-13T11:48:53.180 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.180 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: { 2024-01-13T11:48:53.180 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "fsid": "a3276948-b209-11ee-95ac-87774f69a715", 2024-01-13T11:48:53.180 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "health": { 2024-01-13T11:48:53.180 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-13T11:48:53.180 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "checks": {}, 2024-01-13T11:48:53.180 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "mutes": [] 2024-01-13T11:48:53.180 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:53.181 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-13T11:48:53.181 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "quorum": [ 2024-01-13T11:48:53.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 0 2024-01-13T11:48:53.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ], 2024-01-13T11:48:53.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-13T11:48:53.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "smithi059" 2024-01-13T11:48:53.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ], 2024-01-13T11:48:53.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-01-13T11:48:53.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "monmap": { 2024-01-13T11:48:53.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:48:53.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-13T11:48:53.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-13T11:48:53.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:53.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "osdmap": { 2024-01-13T11:48:53.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:48:53.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-13T11:48:53.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-13T11:48:53.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-13T11:48:53.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-13T11:48:53.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-13T11:48:53.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-13T11:48:53.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:53.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "pgmap": { 2024-01-13T11:48:53.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-13T11:48:53.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-13T11:48:53.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-13T11:48:53.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-13T11:48:53.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-13T11:48:53.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-13T11:48:53.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-13T11:48:53.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-13T11:48:53.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:53.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "fsmap": { 2024-01-13T11:48:53.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:48:53.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-13T11:48:53.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-13T11:48:53.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:53.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-13T11:48:53.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "available": false, 2024-01-13T11:48:53.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-13T11:48:53.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "modules": [ 2024-01-13T11:48:53.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "iostat", 2024-01-13T11:48:53.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "restful" 2024-01-13T11:48:53.187 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ], 2024-01-13T11:48:53.187 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "services": {} 2024-01-13T11:48:53.187 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:53.187 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "servicemap": { 2024-01-13T11:48:53.187 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:48:53.187 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "modified": "2024-01-13T11:48:47.609031+0000", 2024-01-13T11:48:53.187 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "services": {} 2024-01-13T11:48:53.187 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:53.187 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-13T11:48:53.187 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: } 2024-01-13T11:48:53.187 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.190+0000 7f10861ba700 1 -- 172.21.15.59:0/1206731668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10800703d0 msgr2=0x7f108010ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:48:53.187 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.190+0000 7f10861ba700 1 --2- 172.21.15.59:0/1206731668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10800703d0 0x7f108010ca80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f107c00e040 tx=0x7f107c006ee0).stop 2024-01-13T11:48:53.187 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.190+0000 7f10861ba700 1 -- 172.21.15.59:0/1206731668 shutdown_connections 2024-01-13T11:48:53.187 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.190+0000 7f10861ba700 1 --2- 172.21.15.59:0/1206731668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10800703d0 0x7f108010ca80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:48:53.188 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.190+0000 7f10861ba700 1 -- 172.21.15.59:0/1206731668 >> 172.21.15.59:0/1206731668 conn(0x7f10800f6210 msgr2=0x7f10800f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:48:53.188 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.190+0000 7f10861ba700 1 -- 172.21.15.59:0/1206731668 shutdown_connections 2024-01-13T11:48:53.188 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:53.190+0000 7f10861ba700 1 -- 172.21.15.59:0/1206731668 wait complete. 2024-01-13T11:48:53.572 INFO:teuthology.orchestra.run.smithi059.stderr:mgr not available, waiting (1/15)... 2024-01-13T11:48:53.889 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:53 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:48:53.189925+0000 mon.smithi059 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.59:0/1206731668' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-13T11:48:55.574 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-13T11:48:56.005 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.014+0000 7f3c56fef700 1 Processor -- start 2024-01-13T11:48:56.005 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.014+0000 7f3c56fef700 1 -- start start 2024-01-13T11:48:56.005 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.014+0000 7f3c56fef700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c50070430 0x7f3c500fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:48:56.005 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.014+0000 7f3c56fef700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c500fedd0 con 0x7f3c50070430 2024-01-13T11:48:56.005 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.015+0000 7f3c54d8b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c50070430 0x7f3c500fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:48:56.005 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.015+0000 7f3c54d8b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c50070430 0x7f3c500fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59724/0 (socket says 172.21.15.59:59724) 2024-01-13T11:48:56.005 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.015+0000 7f3c54d8b700 1 -- 172.21.15.59:0/1533503287 learned_addr learned my addr 172.21.15.59:0/1533503287 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:48:56.007 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.015+0000 7f3c54d8b700 1 -- 172.21.15.59:0/1533503287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c500fef10 con 0x7f3c50070430 2024-01-13T11:48:56.007 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.015+0000 7f3c54d8b700 1 --2- 172.21.15.59:0/1533503287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c50070430 0x7f3c500fe8e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3c4400b5f0 tx=0x7f3c44006ee0).ready entity=mon.0 client_cookie=7d92e3ee771e4ea server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:48:56.007 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.016+0000 7f3c4f7fe700 1 -- 172.21.15.59:0/1533503287 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c4400d600 con 0x7f3c50070430 2024-01-13T11:48:56.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.016+0000 7f3c4f7fe700 1 -- 172.21.15.59:0/1533503287 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3c4400dbd0 con 0x7f3c50070430 2024-01-13T11:48:56.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.016+0000 7f3c4f7fe700 1 -- 172.21.15.59:0/1533503287 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c44011c00 con 0x7f3c50070430 2024-01-13T11:48:56.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.016+0000 7f3c56fef700 1 -- 172.21.15.59:0/1533503287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c50070430 msgr2=0x7f3c500fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:48:56.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.016+0000 7f3c56fef700 1 --2- 172.21.15.59:0/1533503287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c50070430 0x7f3c500fe8e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3c4400b5f0 tx=0x7f3c44006ee0).stop 2024-01-13T11:48:56.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.016+0000 7f3c56fef700 1 -- 172.21.15.59:0/1533503287 shutdown_connections 2024-01-13T11:48:56.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.016+0000 7f3c56fef700 1 --2- 172.21.15.59:0/1533503287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c50070430 0x7f3c500fe8e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:48:56.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.016+0000 7f3c56fef700 1 -- 172.21.15.59:0/1533503287 >> 172.21.15.59:0/1533503287 conn(0x7f3c500f63c0 msgr2=0x7f3c500f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:48:56.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.016+0000 7f3c56fef700 1 -- 172.21.15.59:0/1533503287 shutdown_connections 2024-01-13T11:48:56.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.016+0000 7f3c56fef700 1 -- 172.21.15.59:0/1533503287 wait complete. 2024-01-13T11:48:56.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.017+0000 7f3c56fef700 1 Processor -- start 2024-01-13T11:48:56.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.017+0000 7f3c56fef700 1 -- start start 2024-01-13T11:48:56.008 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.017+0000 7f3c56fef700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c50070430 0x7f3c5010aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:48:56.009 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.017+0000 7f3c56fef700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5010af30 con 0x7f3c50070430 2024-01-13T11:48:56.009 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.018+0000 7f3c54d8b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c50070430 0x7f3c5010aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:48:56.009 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.018+0000 7f3c54d8b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c50070430 0x7f3c5010aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59738/0 (socket says 172.21.15.59:59738) 2024-01-13T11:48:56.009 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.018+0000 7f3c54d8b700 1 -- 172.21.15.59:0/988550315 learned_addr learned my addr 172.21.15.59:0/988550315 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:48:56.009 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.018+0000 7f3c54d8b700 1 -- 172.21.15.59:0/988550315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4400b200 con 0x7f3c50070430 2024-01-13T11:48:56.009 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.018+0000 7f3c54d8b700 1 --2- 172.21.15.59:0/988550315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c50070430 0x7f3c5010aa40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f3c4400e040 tx=0x7f3c44006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:48:56.010 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.018+0000 7f3c4d7fa700 1 -- 172.21.15.59:0/988550315 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c4400d9f0 con 0x7f3c50070430 2024-01-13T11:48:56.010 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.018+0000 7f3c4d7fa700 1 -- 172.21.15.59:0/988550315 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3c4401a9f0 con 0x7f3c50070430 2024-01-13T11:48:56.010 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.018+0000 7f3c56fef700 1 -- 172.21.15.59:0/988550315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5010b130 con 0x7f3c50070430 2024-01-13T11:48:56.010 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.018+0000 7f3c4d7fa700 1 -- 172.21.15.59:0/988550315 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c4401b8c0 con 0x7f3c50070430 2024-01-13T11:48:56.010 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.018+0000 7f3c56fef700 1 -- 172.21.15.59:0/988550315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5010b540 con 0x7f3c50070430 2024-01-13T11:48:56.010 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.019+0000 7f3c4d7fa700 1 -- 172.21.15.59:0/988550315 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f3c4400f070 con 0x7f3c50070430 2024-01-13T11:48:56.010 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.019+0000 7f3c56fef700 1 -- 172.21.15.59:0/988550315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c50045ad0 con 0x7f3c50070430 2024-01-13T11:48:56.010 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.019+0000 7f3c4d7fa700 1 -- 172.21.15.59:0/988550315 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f3c440257d0 con 0x7f3c50070430 2024-01-13T11:48:56.011 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.021+0000 7f3c4d7fa700 1 -- 172.21.15.59:0/988550315 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f3c44016070 con 0x7f3c50070430 2024-01-13T11:48:56.088 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.092+0000 7f3c56fef700 1 -- 172.21.15.59:0/988550315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3c501047f0 con 0x7f3c50070430 2024-01-13T11:48:56.088 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.093+0000 7f3c4d7fa700 1 -- 172.21.15.59:0/988550315 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f3c4401ba20 con 0x7f3c50070430 2024-01-13T11:48:56.088 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.088 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: { 2024-01-13T11:48:56.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "fsid": "a3276948-b209-11ee-95ac-87774f69a715", 2024-01-13T11:48:56.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "health": { 2024-01-13T11:48:56.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-13T11:48:56.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "checks": {}, 2024-01-13T11:48:56.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "mutes": [] 2024-01-13T11:48:56.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:56.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-13T11:48:56.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "quorum": [ 2024-01-13T11:48:56.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 0 2024-01-13T11:48:56.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ], 2024-01-13T11:48:56.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-13T11:48:56.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "smithi059" 2024-01-13T11:48:56.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ], 2024-01-13T11:48:56.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "quorum_age": 4, 2024-01-13T11:48:56.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "monmap": { 2024-01-13T11:48:56.090 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:48:56.090 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-13T11:48:56.090 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-13T11:48:56.090 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:56.090 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "osdmap": { 2024-01-13T11:48:56.090 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:48:56.090 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-13T11:48:56.090 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-13T11:48:56.090 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-13T11:48:56.090 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-13T11:48:56.090 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-13T11:48:56.090 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-13T11:48:56.090 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:56.090 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "pgmap": { 2024-01-13T11:48:56.091 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-13T11:48:56.091 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-13T11:48:56.091 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-13T11:48:56.091 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-13T11:48:56.091 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-13T11:48:56.091 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-13T11:48:56.091 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-13T11:48:56.091 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-13T11:48:56.091 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:56.091 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "fsmap": { 2024-01-13T11:48:56.091 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:48:56.091 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-13T11:48:56.091 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-13T11:48:56.091 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:56.092 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-13T11:48:56.092 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "available": false, 2024-01-13T11:48:56.092 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-13T11:48:56.092 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "modules": [ 2024-01-13T11:48:56.092 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "iostat", 2024-01-13T11:48:56.092 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "restful" 2024-01-13T11:48:56.092 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ], 2024-01-13T11:48:56.092 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "services": {} 2024-01-13T11:48:56.092 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:56.092 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "servicemap": { 2024-01-13T11:48:56.092 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:48:56.092 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "modified": "2024-01-13T11:48:47.609031+0000", 2024-01-13T11:48:56.092 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "services": {} 2024-01-13T11:48:56.093 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:56.093 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-13T11:48:56.093 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: } 2024-01-13T11:48:56.093 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.093+0000 7f3c56fef700 1 -- 172.21.15.59:0/988550315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c50070430 msgr2=0x7f3c5010aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:48:56.093 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.093+0000 7f3c56fef700 1 --2- 172.21.15.59:0/988550315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c50070430 0x7f3c5010aa40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f3c4400e040 tx=0x7f3c44006ee0).stop 2024-01-13T11:48:56.093 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.094+0000 7f3c56fef700 1 -- 172.21.15.59:0/988550315 shutdown_connections 2024-01-13T11:48:56.093 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.094+0000 7f3c56fef700 1 --2- 172.21.15.59:0/988550315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c50070430 0x7f3c5010aa40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:48:56.093 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.094+0000 7f3c56fef700 1 -- 172.21.15.59:0/988550315 >> 172.21.15.59:0/988550315 conn(0x7f3c500f63c0 msgr2=0x7f3c500f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:48:56.093 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.094+0000 7f3c56fef700 1 -- 172.21.15.59:0/988550315 shutdown_connections 2024-01-13T11:48:56.093 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:56.094+0000 7f3c56fef700 1 -- 172.21.15.59:0/988550315 wait complete. 2024-01-13T11:48:56.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:56 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:48:56.093271+0000 mon.smithi059 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.59:0/988550315' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-13T11:48:56.504 INFO:teuthology.orchestra.run.smithi059.stderr:mgr not available, waiting (2/15)... 2024-01-13T11:48:58.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:48:57.789519+0000 mon.smithi059 (mon.0) 9 : cluster [INF] Activating manager daemon smithi059.hmyldj 2024-01-13T11:48:58.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:48:57.792135+0000 mon.smithi059 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi059.hmyldj(active, starting, since 0.0027384s) 2024-01-13T11:48:58.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:48:57.792827+0000 mon.smithi059 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.59:0/3262035630' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T11:48:58.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:48:57.792977+0000 mon.smithi059 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.59:0/3262035630' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T11:48:58.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:48:57.793161+0000 mon.smithi059 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.59:0/3262035630' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T11:48:58.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:48:57.793356+0000 mon.smithi059 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.59:0/3262035630' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-01-13T11:48:58.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:48:57.793602+0000 mon.smithi059 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.59:0/3262035630' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "mgr metadata", "who": "smithi059.hmyldj", "id": "smithi059.hmyldj"}]: dispatch 2024-01-13T11:48:58.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:48:57.807136+0000 mon.smithi059 (mon.0) 16 : cluster [INF] Manager daemon smithi059.hmyldj is now available 2024-01-13T11:48:58.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:48:57.813582+0000 mon.smithi059 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.59:0/3262035630' entity='mgr.smithi059.hmyldj' 2024-01-13T11:48:58.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:48:57.819493+0000 mon.smithi059 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.59:0/3262035630' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/mirror_snapshot_schedule"}]: dispatch 2024-01-13T11:48:58.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:48:57.820629+0000 mon.smithi059 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.59:0/3262035630' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/trash_purge_schedule"}]: dispatch 2024-01-13T11:48:58.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:48:57.825489+0000 mon.smithi059 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.59:0/3262035630' entity='mgr.smithi059.hmyldj' 2024-01-13T11:48:58.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:48:57.827610+0000 mon.smithi059 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.59:0/3262035630' entity='mgr.smithi059.hmyldj' 2024-01-13T11:48:58.506 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-13T11:48:59.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.457+0000 7f18a4e7a700 1 Processor -- start 2024-01-13T11:48:59.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.457+0000 7f18a4e7a700 1 -- start start 2024-01-13T11:48:59.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.457+0000 7f18a4e7a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18a00703d0 0x7f18a00fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:48:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.457+0000 7f18a4e7a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a00fee30 con 0x7f18a00703d0 2024-01-13T11:48:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.458+0000 7f189e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18a00703d0 0x7f18a00fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:48:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.458+0000 7f189e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18a00703d0 0x7f18a00fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46100/0 (socket says 172.21.15.59:46100) 2024-01-13T11:48:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.458+0000 7f189e59c700 1 -- 172.21.15.59:0/2506683107 learned_addr learned my addr 172.21.15.59:0/2506683107 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:48:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.458+0000 7f189e59c700 1 -- 172.21.15.59:0/2506683107 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a00fef70 con 0x7f18a00703d0 2024-01-13T11:48:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.458+0000 7f189e59c700 1 --2- 172.21.15.59:0/2506683107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18a00703d0 0x7f18a00fe940 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f188800b5f0 tx=0x7f1888006ee0).ready entity=mon.0 client_cookie=de747aae6876369d server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:48:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.458+0000 7f189d59a700 1 -- 172.21.15.59:0/2506683107 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f188800d600 con 0x7f18a00703d0 2024-01-13T11:48:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.458+0000 7f189d59a700 1 -- 172.21.15.59:0/2506683107 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f188800dbd0 con 0x7f18a00703d0 2024-01-13T11:48:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.458+0000 7f189d59a700 1 -- 172.21.15.59:0/2506683107 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1888011c00 con 0x7f18a00703d0 2024-01-13T11:48:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.459+0000 7f18a4e7a700 1 -- 172.21.15.59:0/2506683107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18a00703d0 msgr2=0x7f18a00fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:48:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.459+0000 7f18a4e7a700 1 --2- 172.21.15.59:0/2506683107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18a00703d0 0x7f18a00fe940 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f188800b5f0 tx=0x7f1888006ee0).stop 2024-01-13T11:48:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.459+0000 7f18a4e7a700 1 -- 172.21.15.59:0/2506683107 shutdown_connections 2024-01-13T11:48:59.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.459+0000 7f18a4e7a700 1 --2- 172.21.15.59:0/2506683107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18a00703d0 0x7f18a00fe940 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:48:59.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.459+0000 7f18a4e7a700 1 -- 172.21.15.59:0/2506683107 >> 172.21.15.59:0/2506683107 conn(0x7f18a00f63c0 msgr2=0x7f18a00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:48:59.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.459+0000 7f18a4e7a700 1 -- 172.21.15.59:0/2506683107 shutdown_connections 2024-01-13T11:48:59.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.459+0000 7f18a4e7a700 1 -- 172.21.15.59:0/2506683107 wait complete. 2024-01-13T11:48:59.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.459+0000 7f18a4e7a700 1 Processor -- start 2024-01-13T11:48:59.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.460+0000 7f18a4e7a700 1 -- start start 2024-01-13T11:48:59.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.460+0000 7f18a4e7a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18a00703d0 0x7f18a010a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:48:59.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.460+0000 7f18a4e7a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a010aec0 con 0x7f18a00703d0 2024-01-13T11:48:59.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.460+0000 7f189e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18a00703d0 0x7f18a010a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:48:59.452 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.460+0000 7f189e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18a00703d0 0x7f18a010a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46114/0 (socket says 172.21.15.59:46114) 2024-01-13T11:48:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.460+0000 7f189e59c700 1 -- 172.21.15.59:0/1418927007 learned_addr learned my addr 172.21.15.59:0/1418927007 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:48:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.460+0000 7f189e59c700 1 -- 172.21.15.59:0/1418927007 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f188800b200 con 0x7f18a00703d0 2024-01-13T11:48:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.460+0000 7f189e59c700 1 --2- 172.21.15.59:0/1418927007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18a00703d0 0x7f18a010a9d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f188800b190 tx=0x7f1888006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:48:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.460+0000 7f1896ffd700 1 -- 172.21.15.59:0/1418927007 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f188801b410 con 0x7f18a00703d0 2024-01-13T11:48:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.460+0000 7f1896ffd700 1 -- 172.21.15.59:0/1418927007 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f188801b9e0 con 0x7f18a00703d0 2024-01-13T11:48:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.460+0000 7f18a4e7a700 1 -- 172.21.15.59:0/1418927007 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a010b0c0 con 0x7f18a00703d0 2024-01-13T11:48:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.460+0000 7f1896ffd700 1 -- 172.21.15.59:0/1418927007 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f188801a880 con 0x7f18a00703d0 2024-01-13T11:48:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.460+0000 7f18a4e7a700 1 -- 172.21.15.59:0/1418927007 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a010b4d0 con 0x7f18a00703d0 2024-01-13T11:48:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.461+0000 7f1896ffd700 1 -- 172.21.15.59:0/1418927007 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f188800f070 con 0x7f18a00703d0 2024-01-13T11:48:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.461+0000 7f18a4e7a700 1 -- 172.21.15.59:0/1418927007 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a0045ad0 con 0x7f18a00703d0 2024-01-13T11:48:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.461+0000 7f1896ffd700 1 --2- 172.21.15.59:0/1418927007 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f188002fca0 0x7f1880032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:48:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.461+0000 7f1896ffd700 1 -- 172.21.15.59:0/1418927007 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1888040fe0 con 0x7f18a00703d0 2024-01-13T11:48:59.454 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.462+0000 7f189dd9b700 1 --2- 172.21.15.59:0/1418927007 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f188002fca0 0x7f1880032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:48:59.454 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.462+0000 7f189dd9b700 1 --2- 172.21.15.59:0/1418927007 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f188002fca0 0x7f1880032160 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f18900096a0 tx=0x7f1890006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:48:59.456 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.466+0000 7f1896ffd700 1 -- 172.21.15.59:0/1418927007 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1888019950 con 0x7f18a00703d0 2024-01-13T11:48:59.679 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.682+0000 7f18a4e7a700 1 -- 172.21.15.59:0/1418927007 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f18a00ffb90 con 0x7f18a00703d0 2024-01-13T11:48:59.679 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.682+0000 7f1896ffd700 1 -- 172.21.15.59:0/1418927007 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7f1888016070 con 0x7f18a00703d0 2024-01-13T11:48:59.679 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.679 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: { 2024-01-13T11:48:59.679 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "fsid": "a3276948-b209-11ee-95ac-87774f69a715", 2024-01-13T11:48:59.679 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "health": { 2024-01-13T11:48:59.680 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-13T11:48:59.680 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "checks": {}, 2024-01-13T11:48:59.680 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "mutes": [] 2024-01-13T11:48:59.680 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:59.680 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-13T11:48:59.680 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "quorum": [ 2024-01-13T11:48:59.680 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 0 2024-01-13T11:48:59.680 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ], 2024-01-13T11:48:59.680 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-13T11:48:59.680 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "smithi059" 2024-01-13T11:48:59.680 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ], 2024-01-13T11:48:59.680 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-01-13T11:48:59.681 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "monmap": { 2024-01-13T11:48:59.681 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:48:59.681 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-13T11:48:59.681 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-13T11:48:59.681 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:59.681 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "osdmap": { 2024-01-13T11:48:59.681 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:48:59.681 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-13T11:48:59.681 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-13T11:48:59.681 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-13T11:48:59.681 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-13T11:48:59.681 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-13T11:48:59.682 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-13T11:48:59.682 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:59.682 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "pgmap": { 2024-01-13T11:48:59.682 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-13T11:48:59.682 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-13T11:48:59.682 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-13T11:48:59.682 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-13T11:48:59.682 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-13T11:48:59.682 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-13T11:48:59.682 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-13T11:48:59.682 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-13T11:48:59.682 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:59.683 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "fsmap": { 2024-01-13T11:48:59.683 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:48:59.683 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-13T11:48:59.683 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-13T11:48:59.683 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:59.683 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-13T11:48:59.683 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "available": true, 2024-01-13T11:48:59.683 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-13T11:48:59.683 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "modules": [ 2024-01-13T11:48:59.683 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "iostat", 2024-01-13T11:48:59.683 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "restful" 2024-01-13T11:48:59.683 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ], 2024-01-13T11:48:59.683 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "services": {} 2024-01-13T11:48:59.684 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:59.684 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "servicemap": { 2024-01-13T11:48:59.684 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:48:59.684 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "modified": "2024-01-13T11:48:47.609031+0000", 2024-01-13T11:48:59.684 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "services": {} 2024-01-13T11:48:59.684 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2024-01-13T11:48:59.684 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-13T11:48:59.684 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: } 2024-01-13T11:48:59.684 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.684+0000 7f18a4e7a700 1 -- 172.21.15.59:0/1418927007 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f188002fca0 msgr2=0x7f1880032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:48:59.684 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.684+0000 7f18a4e7a700 1 --2- 172.21.15.59:0/1418927007 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f188002fca0 0x7f1880032160 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f18900096a0 tx=0x7f1890006b40).stop 2024-01-13T11:48:59.684 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.684+0000 7f18a4e7a700 1 -- 172.21.15.59:0/1418927007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18a00703d0 msgr2=0x7f18a010a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:48:59.684 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.684+0000 7f18a4e7a700 1 --2- 172.21.15.59:0/1418927007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18a00703d0 0x7f18a010a9d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f188800b190 tx=0x7f1888006ee0).stop 2024-01-13T11:48:59.685 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.684+0000 7f18a4e7a700 1 -- 172.21.15.59:0/1418927007 shutdown_connections 2024-01-13T11:48:59.685 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.684+0000 7f18a4e7a700 1 --2- 172.21.15.59:0/1418927007 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f188002fca0 0x7f1880032160 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:48:59.685 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.684+0000 7f18a4e7a700 1 --2- 172.21.15.59:0/1418927007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18a00703d0 0x7f18a010a9d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:48:59.685 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.684+0000 7f18a4e7a700 1 -- 172.21.15.59:0/1418927007 >> 172.21.15.59:0/1418927007 conn(0x7f18a00f63c0 msgr2=0x7f18a00f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:48:59.685 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.684+0000 7f18a4e7a700 1 -- 172.21.15.59:0/1418927007 shutdown_connections 2024-01-13T11:48:59.685 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:48:59.684+0000 7f18a4e7a700 1 -- 172.21.15.59:0/1418927007 wait complete. 2024-01-13T11:48:59.956 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:59 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:48:58.795825+0000 mon.smithi059 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi059.hmyldj(active, since 1.00643s) 2024-01-13T11:48:59.956 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:48:59 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:48:59.682958+0000 mon.smithi059 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.59:0/1418927007' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-13T11:49:00.029 INFO:teuthology.orchestra.run.smithi059.stderr:mgr is available 2024-01-13T11:49:00.029 INFO:teuthology.orchestra.run.smithi059.stderr:Enabling cephadm module... 2024-01-13T11:49:00.029 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-01-13T11:49:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.415+0000 7fc7510ae700 1 Processor -- start 2024-01-13T11:49:00.407 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.415+0000 7fc7510ae700 1 -- start start 2024-01-13T11:49:00.407 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.415+0000 7fc7510ae700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc74c0703b0 0x7fc74c100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:00.407 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.415+0000 7fc7510ae700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c0fc750 con 0x7fc74c0703b0 2024-01-13T11:49:00.407 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.415+0000 7fc74ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc74c0703b0 0x7fc74c100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:00.407 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.415+0000 7fc74ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc74c0703b0 0x7fc74c100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46116/0 (socket says 172.21.15.59:46116) 2024-01-13T11:49:00.407 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.415+0000 7fc74ad9d700 1 -- 172.21.15.59:0/1505385738 learned_addr learned my addr 172.21.15.59:0/1505385738 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:00.407 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.416+0000 7fc74ad9d700 1 -- 172.21.15.59:0/1505385738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74c101250 con 0x7fc74c0703b0 2024-01-13T11:49:00.407 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.416+0000 7fc74ad9d700 1 --2- 172.21.15.59:0/1505385738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc74c0703b0 0x7fc74c100d60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fc73c00b5f0 tx=0x7fc73c006ee0).ready entity=mon.0 client_cookie=4b0252eedf1d99a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:00.408 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.416+0000 7fc749d9b700 1 -- 172.21.15.59:0/1505385738 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc73c00d600 con 0x7fc74c0703b0 2024-01-13T11:49:00.408 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.416+0000 7fc749d9b700 1 -- 172.21.15.59:0/1505385738 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc73c00dbd0 con 0x7fc74c0703b0 2024-01-13T11:49:00.408 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.416+0000 7fc749d9b700 1 -- 172.21.15.59:0/1505385738 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc73c011c00 con 0x7fc74c0703b0 2024-01-13T11:49:00.411 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.417+0000 7fc7510ae700 1 -- 172.21.15.59:0/1505385738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc74c0703b0 msgr2=0x7fc74c100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:00.411 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.417+0000 7fc7510ae700 1 --2- 172.21.15.59:0/1505385738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc74c0703b0 0x7fc74c100d60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fc73c00b5f0 tx=0x7fc73c006ee0).stop 2024-01-13T11:49:00.411 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.417+0000 7fc7510ae700 1 -- 172.21.15.59:0/1505385738 shutdown_connections 2024-01-13T11:49:00.411 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.417+0000 7fc7510ae700 1 --2- 172.21.15.59:0/1505385738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc74c0703b0 0x7fc74c100d60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:00.411 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.417+0000 7fc7510ae700 1 -- 172.21.15.59:0/1505385738 >> 172.21.15.59:0/1505385738 conn(0x7fc74c0f63c0 msgr2=0x7fc74c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.417+0000 7fc7510ae700 1 -- 172.21.15.59:0/1505385738 shutdown_connections 2024-01-13T11:49:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.417+0000 7fc7510ae700 1 -- 172.21.15.59:0/1505385738 wait complete. 2024-01-13T11:49:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.418+0000 7fc7510ae700 1 Processor -- start 2024-01-13T11:49:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.418+0000 7fc7510ae700 1 -- start start 2024-01-13T11:49:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.418+0000 7fc7510ae700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc74c0703b0 0x7fc74c10ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.418+0000 7fc7510ae700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c10d1e0 con 0x7fc74c0703b0 2024-01-13T11:49:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.418+0000 7fc74ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc74c0703b0 0x7fc74c10ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.418+0000 7fc74ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc74c0703b0 0x7fc74c10ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46128/0 (socket says 172.21.15.59:46128) 2024-01-13T11:49:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.418+0000 7fc74ad9d700 1 -- 172.21.15.59:0/692524638 learned_addr learned my addr 172.21.15.59:0/692524638 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.418+0000 7fc74ad9d700 1 -- 172.21.15.59:0/692524638 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73c00b200 con 0x7fc74c0703b0 2024-01-13T11:49:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.419+0000 7fc74ad9d700 1 --2- 172.21.15.59:0/692524638 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc74c0703b0 0x7fc74c10ccf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fc73c00e040 tx=0x7fc73c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.419+0000 7fc7437fe700 1 -- 172.21.15.59:0/692524638 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc73c00d9f0 con 0x7fc74c0703b0 2024-01-13T11:49:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.419+0000 7fc7437fe700 1 -- 172.21.15.59:0/692524638 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc73c01a9f0 con 0x7fc74c0703b0 2024-01-13T11:49:00.413 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.419+0000 7fc7437fe700 1 -- 172.21.15.59:0/692524638 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc73c01b8c0 con 0x7fc74c0703b0 2024-01-13T11:49:00.413 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.419+0000 7fc7510ae700 1 -- 172.21.15.59:0/692524638 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc74c10d3e0 con 0x7fc74c0703b0 2024-01-13T11:49:00.413 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.419+0000 7fc7510ae700 1 -- 172.21.15.59:0/692524638 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc74c10d7f0 con 0x7fc74c0703b0 2024-01-13T11:49:00.413 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.420+0000 7fc7437fe700 1 -- 172.21.15.59:0/692524638 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7fc73c00f070 con 0x7fc74c0703b0 2024-01-13T11:49:00.413 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.420+0000 7fc7510ae700 1 -- 172.21.15.59:0/692524638 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc74c045ad0 con 0x7fc74c0703b0 2024-01-13T11:49:00.413 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.420+0000 7fc7437fe700 1 --2- 172.21.15.59:0/692524638 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7fc72c02fd50 0x7fc72c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:00.413 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.420+0000 7fc7437fe700 1 -- 172.21.15.59:0/692524638 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fc73c041190 con 0x7fc74c0703b0 2024-01-13T11:49:00.413 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.420+0000 7fc74a59c700 1 --2- 172.21.15.59:0/692524638 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7fc72c02fd50 0x7fc72c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:00.413 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.420+0000 7fc74a59c700 1 --2- 172.21.15.59:0/692524638 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7fc72c02fd50 0x7fc72c032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc7340096a0 tx=0x7fc734006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:00.414 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.424+0000 7fc7437fe700 1 -- 172.21.15.59:0/692524638 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc73c006a70 con 0x7fc74c0703b0 2024-01-13T11:49:00.610 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.620+0000 7fc7510ae700 1 -- 172.21.15.59:0/692524638 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fc74c101db0 con 0x7fc74c0703b0 2024-01-13T11:49:00.787 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.797+0000 7fc7437fe700 1 -- 172.21.15.59:0/692524638 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fc73c022d80 con 0x7fc74c0703b0 2024-01-13T11:49:00.788 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.798+0000 7fc7437fe700 1 -- 172.21.15.59:0/692524638 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fc73c022310 con 0x7fc74c0703b0 2024-01-13T11:49:00.791 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.800+0000 7fc7510ae700 1 -- 172.21.15.59:0/692524638 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7fc72c02fd50 msgr2=0x7fc72c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:00.791 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.800+0000 7fc7510ae700 1 --2- 172.21.15.59:0/692524638 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7fc72c02fd50 0x7fc72c032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc7340096a0 tx=0x7fc734006b40).stop 2024-01-13T11:49:00.791 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.800+0000 7fc7510ae700 1 -- 172.21.15.59:0/692524638 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc74c0703b0 msgr2=0x7fc74c10ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:00.791 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.800+0000 7fc7510ae700 1 --2- 172.21.15.59:0/692524638 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc74c0703b0 0x7fc74c10ccf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fc73c00e040 tx=0x7fc73c006ee0).stop 2024-01-13T11:49:00.791 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.800+0000 7fc7510ae700 1 -- 172.21.15.59:0/692524638 shutdown_connections 2024-01-13T11:49:00.791 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.800+0000 7fc7510ae700 1 --2- 172.21.15.59:0/692524638 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7fc72c02fd50 0x7fc72c032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:00.791 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.800+0000 7fc7510ae700 1 --2- 172.21.15.59:0/692524638 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc74c0703b0 0x7fc74c10ccf0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:00.791 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.800+0000 7fc7510ae700 1 -- 172.21.15.59:0/692524638 >> 172.21.15.59:0/692524638 conn(0x7fc74c0f63c0 msgr2=0x7fc74c0f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:00.791 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.800+0000 7fc7510ae700 1 -- 172.21.15.59:0/692524638 shutdown_connections 2024-01-13T11:49:00.792 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:00.800+0000 7fc7510ae700 1 -- 172.21.15.59:0/692524638 wait complete. 2024-01-13T11:49:01.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:00 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:48:59.794522+0000 mon.smithi059 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi059.hmyldj(active, since 2s) 2024-01-13T11:49:01.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:00 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:00.620886+0000 mon.smithi059 (mon.0) 25 : audit [INF] from='client.? 172.21.15.59:0/692524638' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-13T11:49:01.162 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-13T11:49:01.522 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.532+0000 7f6241d91700 1 Processor -- start 2024-01-13T11:49:01.524 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.532+0000 7f6241d91700 1 -- start start 2024-01-13T11:49:01.524 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.532+0000 7f6241d91700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f623c0fab80 0x7f623c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:01.524 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.532+0000 7f6241d91700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c0fb490 con 0x7f623c0fab80 2024-01-13T11:49:01.524 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.532+0000 7f623b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f623c0fab80 0x7f623c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:01.524 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.532+0000 7f623b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f623c0fab80 0x7f623c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46158/0 (socket says 172.21.15.59:46158) 2024-01-13T11:49:01.524 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.532+0000 7f623b7fe700 1 -- 172.21.15.59:0/2864020552 learned_addr learned my addr 172.21.15.59:0/2864020552 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:01.525 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.532+0000 7f623b7fe700 1 -- 172.21.15.59:0/2864020552 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623c0fb5d0 con 0x7f623c0fab80 2024-01-13T11:49:01.525 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.533+0000 7f623b7fe700 1 --2- 172.21.15.59:0/2864020552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f623c0fab80 0x7f623c0fafa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f622c00b870 tx=0x7f622c006f70).ready entity=mon.0 client_cookie=d3d62f8ed643561e server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:01.525 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.533+0000 7f623a7fc700 1 -- 172.21.15.59:0/2864020552 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f622c00da20 con 0x7f623c0fab80 2024-01-13T11:49:01.525 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.533+0000 7f623a7fc700 1 -- 172.21.15.59:0/2864020552 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f622c010470 con 0x7f623c0fab80 2024-01-13T11:49:01.525 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.533+0000 7f623a7fc700 1 -- 172.21.15.59:0/2864020552 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f622c00f5b0 con 0x7f623c0fab80 2024-01-13T11:49:01.525 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.533+0000 7f6241d91700 1 -- 172.21.15.59:0/2864020552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f623c0fab80 msgr2=0x7f623c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:01.525 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.533+0000 7f6241d91700 1 --2- 172.21.15.59:0/2864020552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f623c0fab80 0x7f623c0fafa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f622c00b870 tx=0x7f622c006f70).stop 2024-01-13T11:49:01.525 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.533+0000 7f6241d91700 1 -- 172.21.15.59:0/2864020552 shutdown_connections 2024-01-13T11:49:01.525 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.533+0000 7f6241d91700 1 --2- 172.21.15.59:0/2864020552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f623c0fab80 0x7f623c0fafa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:01.525 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.533+0000 7f6241d91700 1 -- 172.21.15.59:0/2864020552 >> 172.21.15.59:0/2864020552 conn(0x7f623c0f63c0 msgr2=0x7f623c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:01.525 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.534+0000 7f6241d91700 1 -- 172.21.15.59:0/2864020552 shutdown_connections 2024-01-13T11:49:01.525 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.534+0000 7f6241d91700 1 -- 172.21.15.59:0/2864020552 wait complete. 2024-01-13T11:49:01.525 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.534+0000 7f6241d91700 1 Processor -- start 2024-01-13T11:49:01.526 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.534+0000 7f6241d91700 1 -- start start 2024-01-13T11:49:01.526 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.534+0000 7f6241d91700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f623c0fab80 0x7f623c10cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:01.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.534+0000 7f6241d91700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c10d070 con 0x7f623c0fab80 2024-01-13T11:49:01.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.535+0000 7f623b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f623c0fab80 0x7f623c10cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:01.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.535+0000 7f623b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f623c0fab80 0x7f623c10cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46174/0 (socket says 172.21.15.59:46174) 2024-01-13T11:49:01.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.535+0000 7f623b7fe700 1 -- 172.21.15.59:0/3078471458 learned_addr learned my addr 172.21.15.59:0/3078471458 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:01.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.535+0000 7f623b7fe700 1 -- 172.21.15.59:0/3078471458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622c00b480 con 0x7f623c0fab80 2024-01-13T11:49:01.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.535+0000 7f623b7fe700 1 --2- 172.21.15.59:0/3078471458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f623c0fab80 0x7f623c10cb80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f622c006800 tx=0x7f622c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:01.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.535+0000 7f6223fff700 1 -- 172.21.15.59:0/3078471458 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f622c00f980 con 0x7f623c0fab80 2024-01-13T11:49:01.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.535+0000 7f6223fff700 1 -- 172.21.15.59:0/3078471458 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f622c019450 con 0x7f623c0fab80 2024-01-13T11:49:01.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.535+0000 7f6241d91700 1 -- 172.21.15.59:0/3078471458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f623c10d270 con 0x7f623c0fab80 2024-01-13T11:49:01.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.535+0000 7f6223fff700 1 -- 172.21.15.59:0/3078471458 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f622c023640 con 0x7f623c0fab80 2024-01-13T11:49:01.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.535+0000 7f6241d91700 1 -- 172.21.15.59:0/3078471458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f623c10d680 con 0x7f623c0fab80 2024-01-13T11:49:01.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.536+0000 7f6223fff700 1 -- 172.21.15.59:0/3078471458 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f622c016030 con 0x7f623c0fab80 2024-01-13T11:49:01.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.536+0000 7f6223fff700 1 --2- 172.21.15.59:0/3078471458 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f622802fd50 0x7f6228032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:01.528 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.536+0000 7f6223fff700 1 -- 172.21.15.59:0/3078471458 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f622c0419d0 con 0x7f623c0fab80 2024-01-13T11:49:01.528 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.536+0000 7f623affd700 1 -- 172.21.15.59:0/3078471458 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f622802fd50 msgr2=0x7f6228032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/70411649 2024-01-13T11:49:01.528 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.537+0000 7f623affd700 1 --2- 172.21.15.59:0/3078471458 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f622802fd50 0x7f6228032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-13T11:49:01.528 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.537+0000 7f6241d91700 1 -- 172.21.15.59:0/3078471458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f623c045ad0 con 0x7f623c0fab80 2024-01-13T11:49:01.530 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.541+0000 7f6223fff700 1 -- 172.21.15.59:0/3078471458 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f622c014070 con 0x7f623c0fab80 2024-01-13T11:49:01.728 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.738+0000 7f623affd700 1 -- 172.21.15.59:0/3078471458 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f622802fd50 msgr2=0x7f6228032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/70411649 2024-01-13T11:49:01.728 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.738+0000 7f623affd700 1 --2- 172.21.15.59:0/3078471458 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f622802fd50 0x7f6228032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-13T11:49:01.728 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.738+0000 7f6241d91700 1 -- 172.21.15.59:0/3078471458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f623c102670 con 0x7f623c0fab80 2024-01-13T11:49:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.738+0000 7f6223fff700 1 -- 172.21.15.59:0/3078471458 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f622c0223a0 con 0x7f623c0fab80 2024-01-13T11:49:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: { 2024-01-13T11:49:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 5, 2024-01-13T11:49:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "available": true, 2024-01-13T11:49:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "active_name": "smithi059.hmyldj", 2024-01-13T11:49:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-13T11:49:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: } 2024-01-13T11:49:01.730 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.740+0000 7f6241d91700 1 -- 172.21.15.59:0/3078471458 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f622802fd50 msgr2=0x7f6228032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-13T11:49:01.731 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.740+0000 7f6241d91700 1 --2- 172.21.15.59:0/3078471458 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f622802fd50 0x7f6228032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-13T11:49:01.731 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.740+0000 7f6241d91700 1 -- 172.21.15.59:0/3078471458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f623c0fab80 msgr2=0x7f623c10cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:01.731 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.740+0000 7f6241d91700 1 --2- 172.21.15.59:0/3078471458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f623c0fab80 0x7f623c10cb80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f622c006800 tx=0x7f622c006f70).stop 2024-01-13T11:49:01.731 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.740+0000 7f6241d91700 1 -- 172.21.15.59:0/3078471458 shutdown_connections 2024-01-13T11:49:01.731 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.740+0000 7f6241d91700 1 --2- 172.21.15.59:0/3078471458 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f622802fd50 0x7f6228032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-13T11:49:01.731 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.740+0000 7f6241d91700 1 --2- 172.21.15.59:0/3078471458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f623c0fab80 0x7f623c10cb80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:01.731 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.740+0000 7f6241d91700 1 -- 172.21.15.59:0/3078471458 >> 172.21.15.59:0/3078471458 conn(0x7f623c0f63c0 msgr2=0x7f623c0ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:01.731 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.740+0000 7f6241d91700 1 -- 172.21.15.59:0/3078471458 shutdown_connections 2024-01-13T11:49:01.731 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:01.740+0000 7f6241d91700 1 -- 172.21.15.59:0/3078471458 wait complete. 2024-01-13T11:49:02.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:01 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:00.799266+0000 mon.smithi059 (mon.0) 26 : audit [INF] from='client.? 172.21.15.59:0/692524638' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-13T11:49:02.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:01 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:00.799325+0000 mon.smithi059 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi059.hmyldj(active, since 3s) 2024-01-13T11:49:02.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:01 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:01.738881+0000 mon.smithi059 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.59:0/3078471458' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-13T11:49:02.082 INFO:teuthology.orchestra.run.smithi059.stderr:Waiting for the mgr to restart... 2024-01-13T11:49:02.083 INFO:teuthology.orchestra.run.smithi059.stderr:Waiting for mgr epoch 5... 2024-01-13T11:49:02.083 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-13T11:49:02.488 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.497+0000 7f35ae863700 1 Processor -- start 2024-01-13T11:49:02.488 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.497+0000 7f35ae863700 1 -- start start 2024-01-13T11:49:02.488 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.497+0000 7f35ae863700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a80fab80 0x7f35a80fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:02.488 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.497+0000 7f35ae863700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a80fb490 con 0x7f35a80fab80 2024-01-13T11:49:02.488 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.497+0000 7f35a7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a80fab80 0x7f35a80fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:02.488 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.497+0000 7f35a7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a80fab80 0x7f35a80fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46178/0 (socket says 172.21.15.59:46178) 2024-01-13T11:49:02.488 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.497+0000 7f35a7fff700 1 -- 172.21.15.59:0/1358279607 learned_addr learned my addr 172.21.15.59:0/1358279607 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:02.488 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.498+0000 7f35a7fff700 1 -- 172.21.15.59:0/1358279607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a80fb5d0 con 0x7f35a80fab80 2024-01-13T11:49:02.488 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.498+0000 7f35a7fff700 1 --2- 172.21.15.59:0/1358279607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a80fab80 0x7f35a80fafa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f359800b5f0 tx=0x7f3598006ee0).ready entity=mon.0 client_cookie=42a948e6653e0f75 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:02.489 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.498+0000 7f35a6ffd700 1 -- 172.21.15.59:0/1358279607 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f359800d600 con 0x7f35a80fab80 2024-01-13T11:49:02.489 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.498+0000 7f35a6ffd700 1 -- 172.21.15.59:0/1358279607 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f359800dbd0 con 0x7f35a80fab80 2024-01-13T11:49:02.489 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.498+0000 7f35a6ffd700 1 -- 172.21.15.59:0/1358279607 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3598011c00 con 0x7f35a80fab80 2024-01-13T11:49:02.490 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.499+0000 7f35ae863700 1 -- 172.21.15.59:0/1358279607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a80fab80 msgr2=0x7f35a80fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:02.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.499+0000 7f35ae863700 1 --2- 172.21.15.59:0/1358279607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a80fab80 0x7f35a80fafa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f359800b5f0 tx=0x7f3598006ee0).stop 2024-01-13T11:49:02.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.499+0000 7f35ae863700 1 -- 172.21.15.59:0/1358279607 shutdown_connections 2024-01-13T11:49:02.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.499+0000 7f35ae863700 1 --2- 172.21.15.59:0/1358279607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a80fab80 0x7f35a80fafa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:02.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.499+0000 7f35ae863700 1 -- 172.21.15.59:0/1358279607 >> 172.21.15.59:0/1358279607 conn(0x7f35a80f63c0 msgr2=0x7f35a80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:02.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.499+0000 7f35ae863700 1 -- 172.21.15.59:0/1358279607 shutdown_connections 2024-01-13T11:49:02.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.499+0000 7f35ae863700 1 -- 172.21.15.59:0/1358279607 wait complete. 2024-01-13T11:49:02.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.499+0000 7f35ae863700 1 Processor -- start 2024-01-13T11:49:02.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.500+0000 7f35ae863700 1 -- start start 2024-01-13T11:49:02.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.500+0000 7f35ae863700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a80fab80 0x7f35a810aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:02.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.500+0000 7f35ae863700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a810aef0 con 0x7f35a80fab80 2024-01-13T11:49:02.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.500+0000 7f35a7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a80fab80 0x7f35a810aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:02.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.500+0000 7f35a7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a80fab80 0x7f35a810aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46192/0 (socket says 172.21.15.59:46192) 2024-01-13T11:49:02.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.500+0000 7f35a7fff700 1 -- 172.21.15.59:0/3268738318 learned_addr learned my addr 172.21.15.59:0/3268738318 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:02.492 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.500+0000 7f35a7fff700 1 -- 172.21.15.59:0/3268738318 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359800b200 con 0x7f35a80fab80 2024-01-13T11:49:02.492 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.500+0000 7f35a7fff700 1 --2- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a80fab80 0x7f35a810aa00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f359800b1d0 tx=0x7f3598006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:02.492 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.501+0000 7f35a4ff9700 1 -- 172.21.15.59:0/3268738318 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f359801b410 con 0x7f35a80fab80 2024-01-13T11:49:02.492 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.501+0000 7f35a4ff9700 1 -- 172.21.15.59:0/3268738318 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f359801b9e0 con 0x7f35a80fab80 2024-01-13T11:49:02.492 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.501+0000 7f35ae863700 1 -- 172.21.15.59:0/3268738318 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a810b0f0 con 0x7f35a80fab80 2024-01-13T11:49:02.492 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.501+0000 7f35a4ff9700 1 -- 172.21.15.59:0/3268738318 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f359801a840 con 0x7f35a80fab80 2024-01-13T11:49:02.493 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.501+0000 7f35ae863700 1 -- 172.21.15.59:0/3268738318 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a810b500 con 0x7f35a80fab80 2024-01-13T11:49:02.493 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.501+0000 7f35a4ff9700 1 -- 172.21.15.59:0/3268738318 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f359800f070 con 0x7f35a80fab80 2024-01-13T11:49:02.493 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.502+0000 7f35a4ff9700 1 --2- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f359402fdb0 0x7f3594032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:02.493 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.502+0000 7f35a4ff9700 1 -- 172.21.15.59:0/3268738318 --> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3594032930 con 0x7f359402fdb0 2024-01-13T11:49:02.493 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.502+0000 7f35a4ff9700 1 -- 172.21.15.59:0/3268738318 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f3598041130 con 0x7f35a80fab80 2024-01-13T11:49:02.493 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.502+0000 7f35a77fe700 1 -- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f359402fdb0 msgr2=0x7f3594032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/70411649 2024-01-13T11:49:02.493 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.502+0000 7f35a77fe700 1 --2- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f359402fdb0 0x7f3594032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-13T11:49:02.692 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.703+0000 7f35a77fe700 1 -- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f359402fdb0 msgr2=0x7f3594032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/70411649 2024-01-13T11:49:02.693 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:02.703+0000 7f35a77fe700 1 --2- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f359402fdb0 0x7f3594032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-13T11:49:03.094 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:03.104+0000 7f35a77fe700 1 -- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f359402fdb0 msgr2=0x7f3594032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/70411649 2024-01-13T11:49:03.094 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:03.104+0000 7f35a77fe700 1 --2- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f359402fdb0 0x7f3594032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-13T11:49:03.894 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:03.905+0000 7f35a77fe700 1 -- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f359402fdb0 msgr2=0x7f3594032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/70411649 2024-01-13T11:49:03.895 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:03.905+0000 7f35a77fe700 1 --2- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f359402fdb0 0x7f3594032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-13T11:49:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:05.506+0000 7f35a77fe700 1 -- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f359402fdb0 msgr2=0x7f3594032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/70411649 2024-01-13T11:49:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:05.506+0000 7f35a77fe700 1 --2- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f359402fdb0 0x7f3594032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-13T11:49:05.997 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:06.007+0000 7f35a4ff9700 1 -- 172.21.15.59:0/3268738318 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f359801a9a0 con 0x7f35a80fab80 2024-01-13T11:49:05.997 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:06.007+0000 7f35a4ff9700 1 -- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f359402fdb0 msgr2=0x7f3594032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-13T11:49:05.997 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:06.007+0000 7f35a4ff9700 1 --2- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f359402fdb0 0x7f3594032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-13T11:49:06.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:06 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:05.953138+0000 mon.smithi059 (mon.0) 29 : cluster [INF] Active manager daemon smithi059.hmyldj restarted 2024-01-13T11:49:06.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:06 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:05.953529+0000 mon.smithi059 (mon.0) 30 : cluster [INF] Activating manager daemon smithi059.hmyldj 2024-01-13T11:49:06.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:06 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:05.957858+0000 mon.smithi059 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-01-13T11:49:07.004 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.014+0000 7f35a4ff9700 1 -- 172.21.15.59:0/3268738318 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f3598042a10 con 0x7f35a80fab80 2024-01-13T11:49:07.004 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.014+0000 7f35a4ff9700 1 --2- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f3594033880 0x7f3594035c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:07.005 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.014+0000 7f35a4ff9700 1 -- 172.21.15.59:0/3268738318 --> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3594032930 con 0x7f3594033880 2024-01-13T11:49:07.005 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.014+0000 7f35a77fe700 1 --2- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f3594033880 0x7f3594035c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:07.005 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.015+0000 7f35a77fe700 1 --2- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f3594033880 0x7f3594035c70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f359c006fd0 tx=0x7f359c006d70).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:07.006 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.016+0000 7f35a4ff9700 1 -- 172.21.15.59:0/3268738318 <== mgr.14114 v2:172.21.15.59:6800/2535313957 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f3594032930 con 0x7f3594033880 2024-01-13T11:49:07.015 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.025+0000 7f35ae863700 1 -- 172.21.15.59:0/3268738318 --> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f35a8059100 con 0x7f3594033880 2024-01-13T11:49:07.015 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.025+0000 7f35a4ff9700 1 -- 172.21.15.59:0/3268738318 <== mgr.14114 v2:172.21.15.59:6800/2535313957 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f35a8059100 con 0x7f3594033880 2024-01-13T11:49:07.015 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: { 2024-01-13T11:49:07.015 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-01-13T11:49:07.016 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "initialized": true 2024-01-13T11:49:07.016 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: } 2024-01-13T11:49:07.016 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.025+0000 7f35ae863700 1 -- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f3594033880 msgr2=0x7f3594035c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:07.017 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.025+0000 7f35ae863700 1 --2- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f3594033880 0x7f3594035c70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f359c006fd0 tx=0x7f359c006d70).stop 2024-01-13T11:49:07.017 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.025+0000 7f35ae863700 1 -- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a80fab80 msgr2=0x7f35a810aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:07.017 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.025+0000 7f35ae863700 1 --2- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a80fab80 0x7f35a810aa00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f359800b1d0 tx=0x7f3598006ee0).stop 2024-01-13T11:49:07.017 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.026+0000 7f35ae863700 1 -- 172.21.15.59:0/3268738318 shutdown_connections 2024-01-13T11:49:07.017 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.026+0000 7f35ae863700 1 --2- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f3594033880 0x7f3594035c70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:07.017 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.026+0000 7f35ae863700 1 --2- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a80fab80 0x7f35a810aa00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:07.018 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.026+0000 7f35ae863700 1 --2- 172.21.15.59:0/3268738318 >> [v2:172.21.15.59:6800/70411649,v1:172.21.15.59:6801/70411649] conn(0x7f359402fdb0 0x7f3594032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-13T11:49:07.018 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.026+0000 7f35ae863700 1 -- 172.21.15.59:0/3268738318 >> 172.21.15.59:0/3268738318 conn(0x7f35a80f63c0 msgr2=0x7f35a8070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:07.018 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.026+0000 7f35ae863700 1 -- 172.21.15.59:0/3268738318 shutdown_connections 2024-01-13T11:49:07.018 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.026+0000 7f35ae863700 1 -- 172.21.15.59:0/3268738318 wait complete. 2024-01-13T11:49:07.323 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:06.008220+0000 mon.smithi059 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi059.hmyldj(active, starting, since 0.0547755s) 2024-01-13T11:49:07.323 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:06.010419+0000 mon.smithi059 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-01-13T11:49:07.323 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:06.011133+0000 mon.smithi059 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "mgr metadata", "who": "smithi059.hmyldj", "id": "smithi059.hmyldj"}]: dispatch 2024-01-13T11:49:07.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:06.012010+0000 mon.smithi059 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T11:49:07.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:06.013799+0000 mon.smithi059 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T11:49:07.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:06.013927+0000 mon.smithi059 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T11:49:07.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:06.016032+0000 mon.smithi059 (mon.0) 38 : cluster [INF] Manager daemon smithi059.hmyldj is now available 2024-01-13T11:49:07.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:06.029220+0000 mon.smithi059 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:07.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:06.031932+0000 mon.smithi059 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:07.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:06.034426+0000 mon.smithi059 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:07.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:06.128040+0000 mon.smithi059 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:07.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:06.128724+0000 mon.smithi059 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:49:07.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:06.130212+0000 mon.smithi059 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:49:07.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:06.132027+0000 mon.smithi059 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:49:07.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:06.142495+0000 mon.smithi059 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/mirror_snapshot_schedule"}]: dispatch 2024-01-13T11:49:07.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:06.143675+0000 mon.smithi059 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/trash_purge_schedule"}]: dispatch 2024-01-13T11:49:07.387 INFO:teuthology.orchestra.run.smithi059.stderr:mgr epoch 5 is available 2024-01-13T11:49:07.387 INFO:teuthology.orchestra.run.smithi059.stderr:Setting orchestrator backend to cephadm... 2024-01-13T11:49:07.387 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-01-13T11:49:07.806 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.816+0000 7f522d433700 1 Processor -- start 2024-01-13T11:49:07.807 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.816+0000 7f522d433700 1 -- start start 2024-01-13T11:49:07.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.817+0000 7f522d433700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52280704a0 0x7f52280fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:07.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.817+0000 7f522d433700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52280fede0 con 0x7f52280704a0 2024-01-13T11:49:07.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.817+0000 7f5226ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52280704a0 0x7f52280fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:07.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.817+0000 7f5226ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52280704a0 0x7f52280fe8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46272/0 (socket says 172.21.15.59:46272) 2024-01-13T11:49:07.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.817+0000 7f5226ffd700 1 -- 172.21.15.59:0/3996524996 learned_addr learned my addr 172.21.15.59:0/3996524996 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:07.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.817+0000 7f5226ffd700 1 -- 172.21.15.59:0/3996524996 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52280fef20 con 0x7f52280704a0 2024-01-13T11:49:07.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.817+0000 7f5226ffd700 1 --2- 172.21.15.59:0/3996524996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52280704a0 0x7f52280fe8f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f521c00b5f0 tx=0x7f521c006ee0).ready entity=mon.0 client_cookie=4797c4395b493382 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:07.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.817+0000 7f5225ffb700 1 -- 172.21.15.59:0/3996524996 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f521c00d600 con 0x7f52280704a0 2024-01-13T11:49:07.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.818+0000 7f5225ffb700 1 -- 172.21.15.59:0/3996524996 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f521c00dbd0 con 0x7f52280704a0 2024-01-13T11:49:07.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.818+0000 7f5225ffb700 1 -- 172.21.15.59:0/3996524996 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f521c011c00 con 0x7f52280704a0 2024-01-13T11:49:07.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.818+0000 7f522d433700 1 -- 172.21.15.59:0/3996524996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52280704a0 msgr2=0x7f52280fe8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:07.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.818+0000 7f522d433700 1 --2- 172.21.15.59:0/3996524996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52280704a0 0x7f52280fe8f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f521c00b5f0 tx=0x7f521c006ee0).stop 2024-01-13T11:49:07.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.818+0000 7f522d433700 1 -- 172.21.15.59:0/3996524996 shutdown_connections 2024-01-13T11:49:07.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.818+0000 7f522d433700 1 --2- 172.21.15.59:0/3996524996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52280704a0 0x7f52280fe8f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:07.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.818+0000 7f522d433700 1 -- 172.21.15.59:0/3996524996 >> 172.21.15.59:0/3996524996 conn(0x7f52280f63b0 msgr2=0x7f52280f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:07.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.818+0000 7f522d433700 1 -- 172.21.15.59:0/3996524996 shutdown_connections 2024-01-13T11:49:07.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.818+0000 7f522d433700 1 -- 172.21.15.59:0/3996524996 wait complete. 2024-01-13T11:49:07.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.819+0000 7f522d433700 1 Processor -- start 2024-01-13T11:49:07.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.819+0000 7f522d433700 1 -- start start 2024-01-13T11:49:07.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.819+0000 7f522d433700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52280704a0 0x7f522810aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:07.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.819+0000 7f522d433700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522810af50 con 0x7f52280704a0 2024-01-13T11:49:07.812 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.820+0000 7f5226ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52280704a0 0x7f522810aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:07.812 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.820+0000 7f5226ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52280704a0 0x7f522810aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46274/0 (socket says 172.21.15.59:46274) 2024-01-13T11:49:07.812 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.820+0000 7f5226ffd700 1 -- 172.21.15.59:0/516818988 learned_addr learned my addr 172.21.15.59:0/516818988 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:07.812 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.820+0000 7f5226ffd700 1 -- 172.21.15.59:0/516818988 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c00b200 con 0x7f52280704a0 2024-01-13T11:49:07.812 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.820+0000 7f5226ffd700 1 --2- 172.21.15.59:0/516818988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52280704a0 0x7f522810aa60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f521c00e040 tx=0x7f521c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:07.812 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.820+0000 7f520f7fe700 1 -- 172.21.15.59:0/516818988 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f521c00d9f0 con 0x7f52280704a0 2024-01-13T11:49:07.812 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.820+0000 7f520f7fe700 1 -- 172.21.15.59:0/516818988 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f521c01a9f0 con 0x7f52280704a0 2024-01-13T11:49:07.812 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.820+0000 7f520f7fe700 1 -- 172.21.15.59:0/516818988 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f521c01b8c0 con 0x7f52280704a0 2024-01-13T11:49:07.812 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.820+0000 7f522d433700 1 -- 172.21.15.59:0/516818988 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522810b150 con 0x7f52280704a0 2024-01-13T11:49:07.812 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.820+0000 7f522d433700 1 -- 172.21.15.59:0/516818988 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f522810b560 con 0x7f52280704a0 2024-01-13T11:49:07.813 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.821+0000 7f520f7fe700 1 -- 172.21.15.59:0/516818988 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f521c00f070 con 0x7f52280704a0 2024-01-13T11:49:07.813 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.821+0000 7f522d433700 1 -- 172.21.15.59:0/516818988 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5228045ad0 con 0x7f52280704a0 2024-01-13T11:49:07.813 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.821+0000 7f520f7fe700 1 --2- 172.21.15.59:0/516818988 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f521402fd10 0x7f52140321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:07.813 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.821+0000 7f520f7fe700 1 -- 172.21.15.59:0/516818988 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f521c041220 con 0x7f52280704a0 2024-01-13T11:49:07.813 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.821+0000 7f52267fc700 1 --2- 172.21.15.59:0/516818988 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f521402fd10 0x7f52140321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:07.813 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.822+0000 7f52267fc700 1 --2- 172.21.15.59:0/516818988 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f521402fd10 0x7f52140321d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f52180096a0 tx=0x7f5218006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:07.815 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:07.826+0000 7f520f7fe700 1 -- 172.21.15.59:0/516818988 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f521c022d70 con 0x7f52280704a0 2024-01-13T11:49:08.004 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.014+0000 7f522d433700 1 -- 172.21.15.59:0/516818988 --> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f522810be40 con 0x7f521402fd10 2024-01-13T11:49:08.009 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.019+0000 7f520f7fe700 1 -- 172.21.15.59:0/516818988 <== mgr.14114 v2:172.21.15.59:6800/2535313957 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f522810be40 con 0x7f521402fd10 2024-01-13T11:49:08.011 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.021+0000 7f520f7fe700 1 -- 172.21.15.59:0/516818988 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f521c0257e0 con 0x7f52280704a0 2024-01-13T11:49:08.012 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.022+0000 7f522d433700 1 -- 172.21.15.59:0/516818988 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f521402fd10 msgr2=0x7f52140321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:08.012 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.022+0000 7f522d433700 1 --2- 172.21.15.59:0/516818988 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f521402fd10 0x7f52140321d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f52180096a0 tx=0x7f5218006b40).stop 2024-01-13T11:49:08.013 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.022+0000 7f522d433700 1 -- 172.21.15.59:0/516818988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52280704a0 msgr2=0x7f522810aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:08.013 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.022+0000 7f522d433700 1 --2- 172.21.15.59:0/516818988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52280704a0 0x7f522810aa60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f521c00e040 tx=0x7f521c006ee0).stop 2024-01-13T11:49:08.013 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.022+0000 7f522d433700 1 -- 172.21.15.59:0/516818988 shutdown_connections 2024-01-13T11:49:08.013 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.022+0000 7f522d433700 1 --2- 172.21.15.59:0/516818988 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f521402fd10 0x7f52140321d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:08.013 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.022+0000 7f522d433700 1 --2- 172.21.15.59:0/516818988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52280704a0 0x7f522810aa60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:08.013 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.022+0000 7f522d433700 1 -- 172.21.15.59:0/516818988 >> 172.21.15.59:0/516818988 conn(0x7f52280f63b0 msgr2=0x7f52280f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:08.014 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.023+0000 7f522d433700 1 -- 172.21.15.59:0/516818988 shutdown_connections 2024-01-13T11:49:08.014 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.024+0000 7f522d433700 1 -- 172.21.15.59:0/516818988 wait complete. 2024-01-13T11:49:08.313 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:08 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:07.016363+0000 mgr.smithi059.hmyldj (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-13T11:49:08.313 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:08 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:07.016904+0000 mon.smithi059 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi059.hmyldj(active, since 1.06345s) 2024-01-13T11:49:08.313 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:08 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:07.025843+0000 mgr.smithi059.hmyldj (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-13T11:49:08.313 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:08 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:08.019609+0000 mon.smithi059 (mon.0) 49 : audit [INF] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:08.353 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-01-13T11:49:08.756 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.763+0000 7f799c3b5700 1 Processor -- start 2024-01-13T11:49:08.756 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.763+0000 7f799c3b5700 1 -- start start 2024-01-13T11:49:08.756 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.764+0000 7f799c3b5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79940fbcb0 0x7f79940fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:08.756 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.764+0000 7f799c3b5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79940fc5c0 con 0x7f79940fbcb0 2024-01-13T11:49:08.756 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.764+0000 7f799a151700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79940fbcb0 0x7f79940fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:08.756 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.764+0000 7f799a151700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79940fbcb0 0x7f79940fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46276/0 (socket says 172.21.15.59:46276) 2024-01-13T11:49:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.764+0000 7f799a151700 1 -- 172.21.15.59:0/268992064 learned_addr learned my addr 172.21.15.59:0/268992064 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.764+0000 7f799a151700 1 -- 172.21.15.59:0/268992064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79940fc700 con 0x7f79940fbcb0 2024-01-13T11:49:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.764+0000 7f799a151700 1 --2- 172.21.15.59:0/268992064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79940fbcb0 0x7f79940fc0d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f798400b5f0 tx=0x7f7984006ee0).ready entity=mon.0 client_cookie=49351fae589283fc server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.765+0000 7f799914f700 1 -- 172.21.15.59:0/268992064 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f798400d600 con 0x7f79940fbcb0 2024-01-13T11:49:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.765+0000 7f799914f700 1 -- 172.21.15.59:0/268992064 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f798400dbd0 con 0x7f79940fbcb0 2024-01-13T11:49:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.765+0000 7f799914f700 1 -- 172.21.15.59:0/268992064 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7984011c00 con 0x7f79940fbcb0 2024-01-13T11:49:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.765+0000 7f799c3b5700 1 -- 172.21.15.59:0/268992064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79940fbcb0 msgr2=0x7f79940fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.765+0000 7f799c3b5700 1 --2- 172.21.15.59:0/268992064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79940fbcb0 0x7f79940fc0d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f798400b5f0 tx=0x7f7984006ee0).stop 2024-01-13T11:49:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.765+0000 7f799c3b5700 1 -- 172.21.15.59:0/268992064 shutdown_connections 2024-01-13T11:49:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.765+0000 7f799c3b5700 1 --2- 172.21.15.59:0/268992064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79940fbcb0 0x7f79940fc0d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:08.758 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.765+0000 7f799c3b5700 1 -- 172.21.15.59:0/268992064 >> 172.21.15.59:0/268992064 conn(0x7f79940f74b0 msgr2=0x7f79940f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:08.758 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.765+0000 7f799c3b5700 1 -- 172.21.15.59:0/268992064 shutdown_connections 2024-01-13T11:49:08.758 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.765+0000 7f799c3b5700 1 -- 172.21.15.59:0/268992064 wait complete. 2024-01-13T11:49:08.758 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.766+0000 7f799c3b5700 1 Processor -- start 2024-01-13T11:49:08.758 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.766+0000 7f799c3b5700 1 -- start start 2024-01-13T11:49:08.758 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.766+0000 7f799c3b5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79940fbcb0 0x7f799410aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:08.759 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.766+0000 7f799c3b5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799410af00 con 0x7f79940fbcb0 2024-01-13T11:49:08.759 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.766+0000 7f799a151700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79940fbcb0 0x7f799410aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:08.759 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.766+0000 7f799a151700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79940fbcb0 0x7f799410aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46288/0 (socket says 172.21.15.59:46288) 2024-01-13T11:49:08.759 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.766+0000 7f799a151700 1 -- 172.21.15.59:0/3549779882 learned_addr learned my addr 172.21.15.59:0/3549779882 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:08.760 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.766+0000 7f799a151700 1 -- 172.21.15.59:0/3549779882 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798400b200 con 0x7f79940fbcb0 2024-01-13T11:49:08.760 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.767+0000 7f799a151700 1 --2- 172.21.15.59:0/3549779882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79940fbcb0 0x7f799410aa10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f798400b1d0 tx=0x7f7984006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:08.760 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.767+0000 7f7982ffd700 1 -- 172.21.15.59:0/3549779882 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f798401b410 con 0x7f79940fbcb0 2024-01-13T11:49:08.761 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.767+0000 7f7982ffd700 1 -- 172.21.15.59:0/3549779882 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f798401b9e0 con 0x7f79940fbcb0 2024-01-13T11:49:08.761 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.767+0000 7f799c3b5700 1 -- 172.21.15.59:0/3549779882 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799410b100 con 0x7f79940fbcb0 2024-01-13T11:49:08.761 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.767+0000 7f7982ffd700 1 -- 172.21.15.59:0/3549779882 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f798401a840 con 0x7f79940fbcb0 2024-01-13T11:49:08.761 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.767+0000 7f799c3b5700 1 -- 172.21.15.59:0/3549779882 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799410b510 con 0x7f79940fbcb0 2024-01-13T11:49:08.761 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.768+0000 7f7982ffd700 1 -- 172.21.15.59:0/3549779882 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f798400f070 con 0x7f79940fbcb0 2024-01-13T11:49:08.761 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.768+0000 7f7982ffd700 1 --2- 172.21.15.59:0/3549779882 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f797c02fdb0 0x7f797c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:08.761 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.768+0000 7f7982ffd700 1 -- 172.21.15.59:0/3549779882 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f7984041210 con 0x7f79940fbcb0 2024-01-13T11:49:08.761 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.768+0000 7f7999950700 1 --2- 172.21.15.59:0/3549779882 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f797c02fdb0 0x7f797c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:08.761 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.768+0000 7f799c3b5700 1 -- 172.21.15.59:0/3549779882 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7994059100 con 0x7f79940fbcb0 2024-01-13T11:49:08.761 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.768+0000 7f7999950700 1 --2- 172.21.15.59:0/3549779882 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f797c02fdb0 0x7f797c032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f79900096a0 tx=0x7f7990006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.772+0000 7f7982ffd700 1 -- 172.21.15.59:0/3549779882 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7984025460 con 0x7f79940fbcb0 2024-01-13T11:49:08.924 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.934+0000 7f799c3b5700 1 -- 172.21.15.59:0/3549779882 --> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f7994104960 con 0x7f797c02fdb0 2024-01-13T11:49:08.924 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.935+0000 7f7982ffd700 1 -- 172.21.15.59:0/3549779882 <== mgr.14114 v2:172.21.15.59:6800/2535313957 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f7994104960 con 0x7f797c02fdb0 2024-01-13T11:49:08.924 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: value unchanged 2024-01-13T11:49:08.927 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.936+0000 7f799c3b5700 1 -- 172.21.15.59:0/3549779882 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f797c02fdb0 msgr2=0x7f797c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:08.927 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.936+0000 7f799c3b5700 1 --2- 172.21.15.59:0/3549779882 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f797c02fdb0 0x7f797c032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f79900096a0 tx=0x7f7990006b40).stop 2024-01-13T11:49:08.927 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.936+0000 7f799c3b5700 1 -- 172.21.15.59:0/3549779882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79940fbcb0 msgr2=0x7f799410aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:08.927 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.936+0000 7f799c3b5700 1 --2- 172.21.15.59:0/3549779882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79940fbcb0 0x7f799410aa10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f798400b1d0 tx=0x7f7984006ee0).stop 2024-01-13T11:49:08.927 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.936+0000 7f799c3b5700 1 -- 172.21.15.59:0/3549779882 shutdown_connections 2024-01-13T11:49:08.927 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.936+0000 7f799c3b5700 1 --2- 172.21.15.59:0/3549779882 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f797c02fdb0 0x7f797c032270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:08.927 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.936+0000 7f799c3b5700 1 --2- 172.21.15.59:0/3549779882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79940fbcb0 0x7f799410aa10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:08.928 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.936+0000 7f799c3b5700 1 -- 172.21.15.59:0/3549779882 >> 172.21.15.59:0/3549779882 conn(0x7f79940f74b0 msgr2=0x7f79940f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:08.928 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.937+0000 7f799c3b5700 1 -- 172.21.15.59:0/3549779882 shutdown_connections 2024-01-13T11:49:08.928 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:08.937+0000 7f799c3b5700 1 -- 172.21.15.59:0/3549779882 wait complete. 2024-01-13T11:49:09.255 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:09 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:08.015704+0000 mgr.smithi059.hmyldj (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:49:09.255 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:09 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:08.023342+0000 mon.smithi059 (mon.0) 50 : cluster [DBG] mgrmap e8: smithi059.hmyldj(active, since 2s) 2024-01-13T11:49:09.255 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:09 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:08.023736+0000 mon.smithi059 (mon.0) 51 : audit [DBG] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:49:09.294 INFO:teuthology.orchestra.run.smithi059.stderr:Generating ssh key... 2024-01-13T11:49:09.294 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-01-13T11:49:09.671 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.678+0000 7f34c77f2700 1 Processor -- start 2024-01-13T11:49:09.671 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.678+0000 7f34c77f2700 1 -- start start 2024-01-13T11:49:09.671 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.678+0000 7f34c77f2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34c00fd340 0x7f34c00fd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:09.671 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.678+0000 7f34c77f2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c00fdc50 con 0x7f34c00fd340 2024-01-13T11:49:09.671 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.678+0000 7f34c558e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34c00fd340 0x7f34c00fd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:09.671 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.678+0000 7f34c558e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34c00fd340 0x7f34c00fd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54454/0 (socket says 172.21.15.59:54454) 2024-01-13T11:49:09.671 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.678+0000 7f34c558e700 1 -- 172.21.15.59:0/373336315 learned_addr learned my addr 172.21.15.59:0/373336315 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.679+0000 7f34c558e700 1 -- 172.21.15.59:0/373336315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c00fdd90 con 0x7f34c00fd340 2024-01-13T11:49:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.679+0000 7f34c558e700 1 --2- 172.21.15.59:0/373336315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34c00fd340 0x7f34c00fd760 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f34b400b5f0 tx=0x7f34b4006ee0).ready entity=mon.0 client_cookie=77835ba9c3e23ffa server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.679+0000 7f34b3fff700 1 -- 172.21.15.59:0/373336315 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34b400d600 con 0x7f34c00fd340 2024-01-13T11:49:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.679+0000 7f34b3fff700 1 -- 172.21.15.59:0/373336315 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f34b400dbd0 con 0x7f34c00fd340 2024-01-13T11:49:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.679+0000 7f34b3fff700 1 -- 172.21.15.59:0/373336315 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34b4011c00 con 0x7f34c00fd340 2024-01-13T11:49:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.679+0000 7f34c77f2700 1 -- 172.21.15.59:0/373336315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34c00fd340 msgr2=0x7f34c00fd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.679+0000 7f34c77f2700 1 --2- 172.21.15.59:0/373336315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34c00fd340 0x7f34c00fd760 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f34b400b5f0 tx=0x7f34b4006ee0).stop 2024-01-13T11:49:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.679+0000 7f34c77f2700 1 -- 172.21.15.59:0/373336315 shutdown_connections 2024-01-13T11:49:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.679+0000 7f34c77f2700 1 --2- 172.21.15.59:0/373336315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34c00fd340 0x7f34c00fd760 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.679+0000 7f34c77f2700 1 -- 172.21.15.59:0/373336315 >> 172.21.15.59:0/373336315 conn(0x7f34c00f5b40 msgr2=0x7f34c00f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.680+0000 7f34c77f2700 1 -- 172.21.15.59:0/373336315 shutdown_connections 2024-01-13T11:49:09.673 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.680+0000 7f34c77f2700 1 -- 172.21.15.59:0/373336315 wait complete. 2024-01-13T11:49:09.673 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.680+0000 7f34c77f2700 1 Processor -- start 2024-01-13T11:49:09.673 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.680+0000 7f34c77f2700 1 -- start start 2024-01-13T11:49:09.673 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.680+0000 7f34c77f2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34c00fd340 0x7f34c010aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:09.673 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.680+0000 7f34c77f2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c010af50 con 0x7f34c00fd340 2024-01-13T11:49:09.673 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.681+0000 7f34c558e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34c00fd340 0x7f34c010aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:09.673 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.681+0000 7f34c558e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34c00fd340 0x7f34c010aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54464/0 (socket says 172.21.15.59:54464) 2024-01-13T11:49:09.673 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.681+0000 7f34c558e700 1 -- 172.21.15.59:0/1727983051 learned_addr learned my addr 172.21.15.59:0/1727983051 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:09.673 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.681+0000 7f34c558e700 1 -- 172.21.15.59:0/1727983051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b400b200 con 0x7f34c00fd340 2024-01-13T11:49:09.673 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.681+0000 7f34c558e700 1 --2- 172.21.15.59:0/1727983051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34c00fd340 0x7f34c010aa60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f34b4002f40 tx=0x7f34b4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:09.673 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.681+0000 7f34b1ffb700 1 -- 172.21.15.59:0/1727983051 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34b401b410 con 0x7f34c00fd340 2024-01-13T11:49:09.674 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.681+0000 7f34b1ffb700 1 -- 172.21.15.59:0/1727983051 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f34b401b9e0 con 0x7f34c00fd340 2024-01-13T11:49:09.674 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.681+0000 7f34c77f2700 1 -- 172.21.15.59:0/1727983051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c010b150 con 0x7f34c00fd340 2024-01-13T11:49:09.674 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.681+0000 7f34b1ffb700 1 -- 172.21.15.59:0/1727983051 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34b401a8b0 con 0x7f34c00fd340 2024-01-13T11:49:09.675 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.681+0000 7f34c77f2700 1 -- 172.21.15.59:0/1727983051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c010b560 con 0x7f34c00fd340 2024-01-13T11:49:09.675 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.682+0000 7f34b1ffb700 1 -- 172.21.15.59:0/1727983051 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f34b400f070 con 0x7f34c00fd340 2024-01-13T11:49:09.675 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.682+0000 7f34c77f2700 1 -- 172.21.15.59:0/1727983051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c0045ad0 con 0x7f34c00fd340 2024-01-13T11:49:09.675 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.682+0000 7f34b1ffb700 1 --2- 172.21.15.59:0/1727983051 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f34a802fdb0 0x7f34a8032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:09.675 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.682+0000 7f34b1ffb700 1 -- 172.21.15.59:0/1727983051 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f34b4041110 con 0x7f34c00fd340 2024-01-13T11:49:09.675 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.683+0000 7f34c4d8d700 1 --2- 172.21.15.59:0/1727983051 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f34a802fdb0 0x7f34a8032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:09.676 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.683+0000 7f34c4d8d700 1 --2- 172.21.15.59:0/1727983051 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f34a802fdb0 0x7f34a8032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f34bc0096a0 tx=0x7f34bc006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:09.676 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.687+0000 7f34b1ffb700 1 -- 172.21.15.59:0/1727983051 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34b401aa10 con 0x7f34c00fd340 2024-01-13T11:49:09.835 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.845+0000 7f34c77f2700 1 -- 172.21.15.59:0/1727983051 --> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f34c010bea0 con 0x7f34a802fdb0 2024-01-13T11:49:09.909 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.918+0000 7f34b1ffb700 1 -- 172.21.15.59:0/1727983051 <== mgr.14114 v2:172.21.15.59:6800/2535313957 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f34c010bea0 con 0x7f34a802fdb0 2024-01-13T11:49:09.910 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.920+0000 7f34c77f2700 1 -- 172.21.15.59:0/1727983051 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f34a802fdb0 msgr2=0x7f34a8032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:09.910 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.920+0000 7f34c77f2700 1 --2- 172.21.15.59:0/1727983051 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f34a802fdb0 0x7f34a8032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f34bc0096a0 tx=0x7f34bc006b40).stop 2024-01-13T11:49:09.911 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.920+0000 7f34c77f2700 1 -- 172.21.15.59:0/1727983051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34c00fd340 msgr2=0x7f34c010aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:09.911 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.920+0000 7f34c77f2700 1 --2- 172.21.15.59:0/1727983051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34c00fd340 0x7f34c010aa60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f34b4002f40 tx=0x7f34b4006ee0).stop 2024-01-13T11:49:09.911 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.920+0000 7f34c77f2700 1 -- 172.21.15.59:0/1727983051 shutdown_connections 2024-01-13T11:49:09.911 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.920+0000 7f34c77f2700 1 --2- 172.21.15.59:0/1727983051 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f34a802fdb0 0x7f34a8032270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:09.911 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.920+0000 7f34c77f2700 1 --2- 172.21.15.59:0/1727983051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34c00fd340 0x7f34c010aa60 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:09.911 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.920+0000 7f34c77f2700 1 -- 172.21.15.59:0/1727983051 >> 172.21.15.59:0/1727983051 conn(0x7f34c00f5b40 msgr2=0x7f34c00ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:09.911 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.921+0000 7f34c77f2700 1 -- 172.21.15.59:0/1727983051 shutdown_connections 2024-01-13T11:49:09.911 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:09.921+0000 7f34c77f2700 1 -- 172.21.15.59:0/1727983051 wait complete. 2024-01-13T11:49:10.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:10 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:08.935393+0000 mgr.smithi059.hmyldj (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:49:10.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:10 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:09.915686+0000 mon.smithi059 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:10.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:10 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:09.918112+0000 mon.smithi059 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:10.578 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-01-13T11:49:11.301 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:11 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cephadm 2024-01-13T11:49:09.846879+0000 mgr.smithi059.hmyldj (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-01-13T11:49:11.913 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.921+0000 7f4f7af8a700 1 Processor -- start 2024-01-13T11:49:11.913 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.922+0000 7f4f7af8a700 1 -- start start 2024-01-13T11:49:11.913 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.922+0000 7f4f7af8a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f74070430 0x7f4f740fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:11.913 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.922+0000 7f4f7af8a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f740fedd0 con 0x7f4f74070430 2024-01-13T11:49:11.914 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.922+0000 7f4f78d26700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f74070430 0x7f4f740fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:11.914 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.922+0000 7f4f78d26700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f74070430 0x7f4f740fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54476/0 (socket says 172.21.15.59:54476) 2024-01-13T11:49:11.914 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.922+0000 7f4f78d26700 1 -- 172.21.15.59:0/1624501169 learned_addr learned my addr 172.21.15.59:0/1624501169 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:11.914 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.922+0000 7f4f78d26700 1 -- 172.21.15.59:0/1624501169 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f740fef10 con 0x7f4f74070430 2024-01-13T11:49:11.916 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.922+0000 7f4f78d26700 1 --2- 172.21.15.59:0/1624501169 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f74070430 0x7f4f740fe8e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f4f6800b5f0 tx=0x7f4f68006ee0).ready entity=mon.0 client_cookie=f3173291a398de0b server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.922+0000 7f4f737fe700 1 -- 172.21.15.59:0/1624501169 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f6800d600 con 0x7f4f74070430 2024-01-13T11:49:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.923+0000 7f4f737fe700 1 -- 172.21.15.59:0/1624501169 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4f6800dbd0 con 0x7f4f74070430 2024-01-13T11:49:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.923+0000 7f4f737fe700 1 -- 172.21.15.59:0/1624501169 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f68011c00 con 0x7f4f74070430 2024-01-13T11:49:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.923+0000 7f4f7af8a700 1 -- 172.21.15.59:0/1624501169 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f74070430 msgr2=0x7f4f740fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.923+0000 7f4f7af8a700 1 --2- 172.21.15.59:0/1624501169 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f74070430 0x7f4f740fe8e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f4f6800b5f0 tx=0x7f4f68006ee0).stop 2024-01-13T11:49:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.923+0000 7f4f7af8a700 1 -- 172.21.15.59:0/1624501169 shutdown_connections 2024-01-13T11:49:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.923+0000 7f4f7af8a700 1 --2- 172.21.15.59:0/1624501169 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f74070430 0x7f4f740fe8e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.923+0000 7f4f7af8a700 1 -- 172.21.15.59:0/1624501169 >> 172.21.15.59:0/1624501169 conn(0x7f4f740f63c0 msgr2=0x7f4f740f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.924+0000 7f4f7af8a700 1 -- 172.21.15.59:0/1624501169 shutdown_connections 2024-01-13T11:49:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.924+0000 7f4f7af8a700 1 -- 172.21.15.59:0/1624501169 wait complete. 2024-01-13T11:49:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.924+0000 7f4f7af8a700 1 Processor -- start 2024-01-13T11:49:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.924+0000 7f4f7af8a700 1 -- start start 2024-01-13T11:49:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.925+0000 7f4f7af8a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f74070430 0x7f4f740fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.925+0000 7f4f7af8a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f740fa7e0 con 0x7f4f74070430 2024-01-13T11:49:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.925+0000 7f4f78d26700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f74070430 0x7f4f740fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.925+0000 7f4f78d26700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f74070430 0x7f4f740fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54488/0 (socket says 172.21.15.59:54488) 2024-01-13T11:49:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.925+0000 7f4f78d26700 1 -- 172.21.15.59:0/4073335344 learned_addr learned my addr 172.21.15.59:0/4073335344 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.925+0000 7f4f78d26700 1 -- 172.21.15.59:0/4073335344 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6800b200 con 0x7f4f74070430 2024-01-13T11:49:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.925+0000 7f4f78d26700 1 --2- 172.21.15.59:0/4073335344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f74070430 0x7f4f740fc360 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f4f6800b190 tx=0x7f4f68006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.925+0000 7f4f717fa700 1 -- 172.21.15.59:0/4073335344 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f6801b410 con 0x7f4f74070430 2024-01-13T11:49:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.925+0000 7f4f7af8a700 1 -- 172.21.15.59:0/4073335344 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f740fa980 con 0x7f4f74070430 2024-01-13T11:49:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.925+0000 7f4f717fa700 1 -- 172.21.15.59:0/4073335344 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4f6801b9e0 con 0x7f4f74070430 2024-01-13T11:49:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.925+0000 7f4f717fa700 1 -- 172.21.15.59:0/4073335344 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f6801a880 con 0x7f4f74070430 2024-01-13T11:49:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.925+0000 7f4f7af8a700 1 -- 172.21.15.59:0/4073335344 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f740fad90 con 0x7f4f74070430 2024-01-13T11:49:11.919 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.927+0000 7f4f7af8a700 1 -- 172.21.15.59:0/4073335344 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f74045ad0 con 0x7f4f74070430 2024-01-13T11:49:11.919 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.927+0000 7f4f717fa700 1 -- 172.21.15.59:0/4073335344 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4f6800f070 con 0x7f4f74070430 2024-01-13T11:49:11.919 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.927+0000 7f4f717fa700 1 --2- 172.21.15.59:0/4073335344 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f4f5802fd50 0x7f4f58032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:11.920 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.927+0000 7f4f717fa700 1 -- 172.21.15.59:0/4073335344 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f4f68041290 con 0x7f4f74070430 2024-01-13T11:49:11.920 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.927+0000 7f4f73fff700 1 --2- 172.21.15.59:0/4073335344 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f4f5802fd50 0x7f4f58032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:11.920 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.927+0000 7f4f73fff700 1 --2- 172.21.15.59:0/4073335344 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f4f5802fd50 0x7f4f58032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f4f600096a0 tx=0x7f4f60006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:11.922 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:11.931+0000 7f4f717fa700 1 -- 172.21.15.59:0/4073335344 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4f68019cd0 con 0x7f4f74070430 2024-01-13T11:49:12.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:12.091+0000 7f4f7af8a700 1 -- 172.21.15.59:0/4073335344 --> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f740ffd10 con 0x7f4f5802fd50 2024-01-13T11:49:12.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:12.092+0000 7f4f717fa700 1 -- 172.21.15.59:0/4073335344 <== mgr.14114 v2:172.21.15.59:6800/2535313957 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f4f740ffd10 con 0x7f4f5802fd50 2024-01-13T11:49:12.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXLWI4l1k5ZexsEczQd4VVY2MeYhyIbfX57PRHiP8eeGzVCLZyujgbltA277plW7z5xppPUinicJyBGQXC2PkbStIzfjKazLbIHXgWh1GAw142UsyqrvZDAs5Mfd89DHt4u1IZjb7hgHIqfvGIy8vYPEXhtKjzcM+SQ3KLrh0lCqFwfRNB0JFSiLljRrxxSvamNwN3A4Uhl6Jq+dt/zx1juFT2btKeDuHuIpzzj7ztDIvCbOoI6YzznnnNx/qqxwv+2V4eJLBQ9EJWN0VTre/IJuSGc6RZxt1qmnv8+v469njUyXuoOariIH1rT2+vbYuswkmHIjkmkiB1ytzsRlbUYIfpTHLWR9dPkQq0Xh2861XXcWyMrV9+kdjYKuvraOQxzc+daTeg90aswiUvUbox9fgHvaR/SKPkJon0PKXyV+lYJ3Yh0QUj0usBYHXt6ZREhfwd6b/am3b2dBToOPh+uxj55D7e90ji2/rVybQ4OhMb3lzvyygMIrzAqQzA2eU= ceph-a3276948-b209-11ee-95ac-87774f69a715 2024-01-13T11:49:12.085 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:12.093+0000 7f4f7af8a700 1 -- 172.21.15.59:0/4073335344 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f4f5802fd50 msgr2=0x7f4f58032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:12.085 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:12.093+0000 7f4f7af8a700 1 --2- 172.21.15.59:0/4073335344 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f4f5802fd50 0x7f4f58032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f4f600096a0 tx=0x7f4f60006b40).stop 2024-01-13T11:49:12.085 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:12.093+0000 7f4f7af8a700 1 -- 172.21.15.59:0/4073335344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f74070430 msgr2=0x7f4f740fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:12.085 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:12.093+0000 7f4f7af8a700 1 --2- 172.21.15.59:0/4073335344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f74070430 0x7f4f740fc360 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f4f6800b190 tx=0x7f4f68006ee0).stop 2024-01-13T11:49:12.085 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:12.093+0000 7f4f7af8a700 1 -- 172.21.15.59:0/4073335344 shutdown_connections 2024-01-13T11:49:12.085 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:12.093+0000 7f4f7af8a700 1 --2- 172.21.15.59:0/4073335344 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f4f5802fd50 0x7f4f58032210 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:12.085 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:12.093+0000 7f4f7af8a700 1 --2- 172.21.15.59:0/4073335344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f74070430 0x7f4f740fc360 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:12.085 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:12.093+0000 7f4f7af8a700 1 -- 172.21.15.59:0/4073335344 >> 172.21.15.59:0/4073335344 conn(0x7f4f740f63c0 msgr2=0x7f4f740f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:12.085 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:12.094+0000 7f4f7af8a700 1 -- 172.21.15.59:0/4073335344 shutdown_connections 2024-01-13T11:49:12.085 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:12.094+0000 7f4f7af8a700 1 -- 172.21.15.59:0/4073335344 wait complete. 2024-01-13T11:49:13.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:12 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:12.093373+0000 mgr.smithi059.hmyldj (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:49:14.540 INFO:teuthology.orchestra.run.smithi059.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-13T11:49:14.540 INFO:teuthology.orchestra.run.smithi059.stderr:Adding key to root@localhost authorized_keys... 2024-01-13T11:49:14.541 INFO:teuthology.orchestra.run.smithi059.stderr:Adding host smithi059... 2024-01-13T11:49:14.541 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi059 172.21.15.59 2024-01-13T11:49:15.583 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.569+0000 7f7162746700 1 Processor -- start 2024-01-13T11:49:15.584 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.569+0000 7f7162746700 1 -- start start 2024-01-13T11:49:15.584 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.570+0000 7f7162746700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f715c0fab80 0x7f715c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:15.584 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.570+0000 7f7162746700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c0fb490 con 0x7f715c0fab80 2024-01-13T11:49:15.584 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.570+0000 7f715bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f715c0fab80 0x7f715c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:15.584 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.570+0000 7f715bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f715c0fab80 0x7f715c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54502/0 (socket says 172.21.15.59:54502) 2024-01-13T11:49:15.584 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.570+0000 7f715bfff700 1 -- 172.21.15.59:0/649501046 learned_addr learned my addr 172.21.15.59:0/649501046 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:15.584 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.570+0000 7f715bfff700 1 -- 172.21.15.59:0/649501046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715c0fb5d0 con 0x7f715c0fab80 2024-01-13T11:49:15.584 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.570+0000 7f715bfff700 1 --2- 172.21.15.59:0/649501046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f715c0fab80 0x7f715c0fafa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f714c00b5f0 tx=0x7f714c006ee0).ready entity=mon.0 client_cookie=44ab182f66d9eec server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:15.584 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.571+0000 7f715affd700 1 -- 172.21.15.59:0/649501046 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f714c00d600 con 0x7f715c0fab80 2024-01-13T11:49:15.584 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.571+0000 7f715affd700 1 -- 172.21.15.59:0/649501046 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f714c00dbd0 con 0x7f715c0fab80 2024-01-13T11:49:15.584 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.571+0000 7f715affd700 1 -- 172.21.15.59:0/649501046 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f714c011c00 con 0x7f715c0fab80 2024-01-13T11:49:15.585 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.571+0000 7f7162746700 1 -- 172.21.15.59:0/649501046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f715c0fab80 msgr2=0x7f715c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:15.585 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.571+0000 7f7162746700 1 --2- 172.21.15.59:0/649501046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f715c0fab80 0x7f715c0fafa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f714c00b5f0 tx=0x7f714c006ee0).stop 2024-01-13T11:49:15.585 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.571+0000 7f7162746700 1 -- 172.21.15.59:0/649501046 shutdown_connections 2024-01-13T11:49:15.585 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.571+0000 7f7162746700 1 --2- 172.21.15.59:0/649501046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f715c0fab80 0x7f715c0fafa0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:15.585 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.571+0000 7f7162746700 1 -- 172.21.15.59:0/649501046 >> 172.21.15.59:0/649501046 conn(0x7f715c0f63c0 msgr2=0x7f715c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:15.587 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.571+0000 7f7162746700 1 -- 172.21.15.59:0/649501046 shutdown_connections 2024-01-13T11:49:15.587 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.571+0000 7f7162746700 1 -- 172.21.15.59:0/649501046 wait complete. 2024-01-13T11:49:15.588 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.572+0000 7f7162746700 1 Processor -- start 2024-01-13T11:49:15.588 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.572+0000 7f7162746700 1 -- start start 2024-01-13T11:49:15.588 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.572+0000 7f7162746700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f715c0fab80 0x7f715c109930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:15.588 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.572+0000 7f7162746700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c109e20 con 0x7f715c0fab80 2024-01-13T11:49:15.588 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.572+0000 7f715bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f715c0fab80 0x7f715c109930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:15.588 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.572+0000 7f715bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f715c0fab80 0x7f715c109930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54518/0 (socket says 172.21.15.59:54518) 2024-01-13T11:49:15.588 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.572+0000 7f715bfff700 1 -- 172.21.15.59:0/3425685601 learned_addr learned my addr 172.21.15.59:0/3425685601 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:15.588 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.572+0000 7f715bfff700 1 -- 172.21.15.59:0/3425685601 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714c00b200 con 0x7f715c0fab80 2024-01-13T11:49:15.588 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.572+0000 7f715bfff700 1 --2- 172.21.15.59:0/3425685601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f715c0fab80 0x7f715c109930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f714c00e040 tx=0x7f714c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:15.588 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.573+0000 7f7158ff9700 1 -- 172.21.15.59:0/3425685601 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f714c01b410 con 0x7f715c0fab80 2024-01-13T11:49:15.588 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.573+0000 7f7158ff9700 1 -- 172.21.15.59:0/3425685601 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f714c01b9e0 con 0x7f715c0fab80 2024-01-13T11:49:15.588 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.573+0000 7f7162746700 1 -- 172.21.15.59:0/3425685601 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f715c10a020 con 0x7f715c0fab80 2024-01-13T11:49:15.588 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.573+0000 7f7158ff9700 1 -- 172.21.15.59:0/3425685601 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f714c01a820 con 0x7f715c0fab80 2024-01-13T11:49:15.589 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.573+0000 7f7162746700 1 -- 172.21.15.59:0/3425685601 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f715c10a430 con 0x7f715c0fab80 2024-01-13T11:49:15.589 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.573+0000 7f7158ff9700 1 -- 172.21.15.59:0/3425685601 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f714c00f070 con 0x7f715c0fab80 2024-01-13T11:49:15.589 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.574+0000 7f7158ff9700 1 --2- 172.21.15.59:0/3425685601 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f714802fdb0 0x7f7148032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:15.589 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.574+0000 7f7162746700 1 -- 172.21.15.59:0/3425685601 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f715c045ad0 con 0x7f715c0fab80 2024-01-13T11:49:15.589 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.574+0000 7f7158ff9700 1 -- 172.21.15.59:0/3425685601 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f714c041430 con 0x7f715c0fab80 2024-01-13T11:49:15.589 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.574+0000 7f715b7fe700 1 --2- 172.21.15.59:0/3425685601 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f714802fdb0 0x7f7148032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:15.589 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.574+0000 7f715b7fe700 1 --2- 172.21.15.59:0/3425685601 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f714802fdb0 0x7f7148032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f71500096a0 tx=0x7f7150006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:15.589 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.578+0000 7f7158ff9700 1 -- 172.21.15.59:0/3425685601 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f714c06c2b0 con 0x7f715c0fab80 2024-01-13T11:49:15.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:15.762+0000 7f7162746700 1 -- 172.21.15.59:0/3425685601 --> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi059", "addr": "172.21.15.59", "target": ["mon-mgr", ""]}) v1 -- 0x7f715c10ad10 con 0x7f714802fdb0 2024-01-13T11:49:16.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:15 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:15.764871+0000 mgr.smithi059.hmyldj (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi059", "addr": "172.21.15.59", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:49:16.426 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: Added host 'smithi059' 2024-01-13T11:49:16.426 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:16.434+0000 7f7158ff9700 1 -- 172.21.15.59:0/3425685601 <== mgr.14114 v2:172.21.15.59:6800/2535313957 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f715c10ad10 con 0x7f714802fdb0 2024-01-13T11:49:16.427 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:16.436+0000 7f7162746700 1 -- 172.21.15.59:0/3425685601 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f714802fdb0 msgr2=0x7f7148032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:16.427 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:16.436+0000 7f7162746700 1 --2- 172.21.15.59:0/3425685601 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f714802fdb0 0x7f7148032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f71500096a0 tx=0x7f7150006b40).stop 2024-01-13T11:49:16.428 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:16.436+0000 7f7162746700 1 -- 172.21.15.59:0/3425685601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f715c0fab80 msgr2=0x7f715c109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:16.428 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:16.436+0000 7f7162746700 1 --2- 172.21.15.59:0/3425685601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f715c0fab80 0x7f715c109930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f714c00e040 tx=0x7f714c006ee0).stop 2024-01-13T11:49:16.428 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:16.437+0000 7f7162746700 1 -- 172.21.15.59:0/3425685601 shutdown_connections 2024-01-13T11:49:16.428 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:16.437+0000 7f7162746700 1 --2- 172.21.15.59:0/3425685601 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f714802fdb0 0x7f7148032270 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:16.428 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:16.437+0000 7f7162746700 1 --2- 172.21.15.59:0/3425685601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f715c0fab80 0x7f715c109930 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:16.428 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:16.437+0000 7f7162746700 1 -- 172.21.15.59:0/3425685601 >> 172.21.15.59:0/3425685601 conn(0x7f715c0f63c0 msgr2=0x7f715c0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:16.428 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:16.437+0000 7f7162746700 1 -- 172.21.15.59:0/3425685601 shutdown_connections 2024-01-13T11:49:16.428 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:16.437+0000 7f7162746700 1 -- 172.21.15.59:0/3425685601 wait complete. 2024-01-13T11:49:17.889 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:17 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:16.435005+0000 mon.smithi059 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:17.889 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:17 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cephadm 2024-01-13T11:49:16.435300+0000 mgr.smithi059.hmyldj (mgr.14114) 9 : cephadm [INF] Added host smithi059 2024-01-13T11:49:17.889 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:17 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:16.435583+0000 mon.smithi059 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:49:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:Deploying mon service with default placement... 2024-01-13T11:49:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-01-13T11:49:18.695 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.702+0000 7f9e6147a700 1 Processor -- start 2024-01-13T11:49:18.695 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.702+0000 7f9e6147a700 1 -- start start 2024-01-13T11:49:18.695 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.702+0000 7f9e6147a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e5c0fab90 0x7f9e5c0fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:18.695 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.702+0000 7f9e6147a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c0fb4a0 con 0x7f9e5c0fab90 2024-01-13T11:49:18.695 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.703+0000 7f9e5affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e5c0fab90 0x7f9e5c0fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:18.695 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.703+0000 7f9e5affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e5c0fab90 0x7f9e5c0fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54522/0 (socket says 172.21.15.59:54522) 2024-01-13T11:49:18.696 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.703+0000 7f9e5affd700 1 -- 172.21.15.59:0/1621413520 learned_addr learned my addr 172.21.15.59:0/1621413520 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:18.699 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.703+0000 7f9e5affd700 1 -- 172.21.15.59:0/1621413520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c0fb5e0 con 0x7f9e5c0fab90 2024-01-13T11:49:18.699 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.703+0000 7f9e5affd700 1 --2- 172.21.15.59:0/1621413520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e5c0fab90 0x7f9e5c0fafb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9e4c00b5f0 tx=0x7f9e4c006ee0).ready entity=mon.0 client_cookie=b8b8150797fbb555 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:18.699 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.703+0000 7f9e59ffb700 1 -- 172.21.15.59:0/1621413520 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e4c00d600 con 0x7f9e5c0fab90 2024-01-13T11:49:18.699 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.703+0000 7f9e59ffb700 1 -- 172.21.15.59:0/1621413520 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9e4c00dbd0 con 0x7f9e5c0fab90 2024-01-13T11:49:18.699 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.703+0000 7f9e59ffb700 1 -- 172.21.15.59:0/1621413520 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e4c011c00 con 0x7f9e5c0fab90 2024-01-13T11:49:18.699 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.704+0000 7f9e6147a700 1 -- 172.21.15.59:0/1621413520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e5c0fab90 msgr2=0x7f9e5c0fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:18.699 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.704+0000 7f9e6147a700 1 --2- 172.21.15.59:0/1621413520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e5c0fab90 0x7f9e5c0fafb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9e4c00b5f0 tx=0x7f9e4c006ee0).stop 2024-01-13T11:49:18.699 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.704+0000 7f9e6147a700 1 -- 172.21.15.59:0/1621413520 shutdown_connections 2024-01-13T11:49:18.699 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.704+0000 7f9e6147a700 1 --2- 172.21.15.59:0/1621413520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e5c0fab90 0x7f9e5c0fafb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:18.700 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.704+0000 7f9e6147a700 1 -- 172.21.15.59:0/1621413520 >> 172.21.15.59:0/1621413520 conn(0x7f9e5c0f63b0 msgr2=0x7f9e5c0f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:18.700 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.704+0000 7f9e6147a700 1 -- 172.21.15.59:0/1621413520 shutdown_connections 2024-01-13T11:49:18.700 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.704+0000 7f9e6147a700 1 -- 172.21.15.59:0/1621413520 wait complete. 2024-01-13T11:49:18.702 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.705+0000 7f9e6147a700 1 Processor -- start 2024-01-13T11:49:18.702 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.705+0000 7f9e6147a700 1 -- start start 2024-01-13T11:49:18.702 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.705+0000 7f9e6147a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e5c0fab90 0x7f9e5c10d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:18.702 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.705+0000 7f9e6147a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c0704e0 con 0x7f9e5c0fab90 2024-01-13T11:49:18.702 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.705+0000 7f9e5affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e5c0fab90 0x7f9e5c10d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:18.703 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.705+0000 7f9e5affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e5c0fab90 0x7f9e5c10d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54538/0 (socket says 172.21.15.59:54538) 2024-01-13T11:49:18.703 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.705+0000 7f9e5affd700 1 -- 172.21.15.59:0/886817003 learned_addr learned my addr 172.21.15.59:0/886817003 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:18.703 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.705+0000 7f9e5affd700 1 -- 172.21.15.59:0/886817003 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e4c00b200 con 0x7f9e5c0fab90 2024-01-13T11:49:18.703 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.706+0000 7f9e5affd700 1 --2- 172.21.15.59:0/886817003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e5c0fab90 0x7f9e5c10d740 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9e4c00e040 tx=0x7f9e4c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:18.703 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.706+0000 7f9e437fe700 1 -- 172.21.15.59:0/886817003 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e4c01b410 con 0x7f9e5c0fab90 2024-01-13T11:49:18.703 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.706+0000 7f9e437fe700 1 -- 172.21.15.59:0/886817003 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9e4c01b9e0 con 0x7f9e5c0fab90 2024-01-13T11:49:18.703 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.706+0000 7f9e437fe700 1 -- 172.21.15.59:0/886817003 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e4c01a820 con 0x7f9e5c0fab90 2024-01-13T11:49:18.703 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.706+0000 7f9e6147a700 1 -- 172.21.15.59:0/886817003 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5c0706e0 con 0x7f9e5c0fab90 2024-01-13T11:49:18.703 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.706+0000 7f9e6147a700 1 -- 172.21.15.59:0/886817003 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e5c10ddf0 con 0x7f9e5c0fab90 2024-01-13T11:49:18.703 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.707+0000 7f9e6147a700 1 -- 172.21.15.59:0/886817003 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e5c045ad0 con 0x7f9e5c0fab90 2024-01-13T11:49:18.703 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.707+0000 7f9e437fe700 1 -- 172.21.15.59:0/886817003 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f9e4c00f070 con 0x7f9e5c0fab90 2024-01-13T11:49:18.703 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.707+0000 7f9e437fe700 1 --2- 172.21.15.59:0/886817003 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f9e4802fdb0 0x7f9e48032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:18.703 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.708+0000 7f9e437fe700 1 -- 172.21.15.59:0/886817003 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f9e4c041ac0 con 0x7f9e5c0fab90 2024-01-13T11:49:18.704 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.708+0000 7f9e5a7fc700 1 --2- 172.21.15.59:0/886817003 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f9e4802fdb0 0x7f9e48032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:18.704 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.708+0000 7f9e5a7fc700 1 --2- 172.21.15.59:0/886817003 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f9e4802fdb0 0x7f9e48032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f9e500096a0 tx=0x7f9e50006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:18.704 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.711+0000 7f9e437fe700 1 -- 172.21.15.59:0/886817003 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e4c019300 con 0x7f9e5c0fab90 2024-01-13T11:49:18.889 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.899+0000 7f9e6147a700 1 -- 172.21.15.59:0/886817003 --> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e5c057f10 con 0x7f9e4802fdb0 2024-01-13T11:49:18.893 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.903+0000 7f9e437fe700 1 -- 172.21.15.59:0/886817003 <== mgr.14114 v2:172.21.15.59:6800/2535313957 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9e5c057f10 con 0x7f9e4802fdb0 2024-01-13T11:49:18.894 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: Scheduled mon update... 2024-01-13T11:49:18.897 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.905+0000 7f9e6147a700 1 -- 172.21.15.59:0/886817003 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f9e4802fdb0 msgr2=0x7f9e48032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:18.897 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.905+0000 7f9e6147a700 1 --2- 172.21.15.59:0/886817003 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f9e4802fdb0 0x7f9e48032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f9e500096a0 tx=0x7f9e50006b40).stop 2024-01-13T11:49:18.897 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.905+0000 7f9e6147a700 1 -- 172.21.15.59:0/886817003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e5c0fab90 msgr2=0x7f9e5c10d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:18.897 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.905+0000 7f9e6147a700 1 --2- 172.21.15.59:0/886817003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e5c0fab90 0x7f9e5c10d740 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9e4c00e040 tx=0x7f9e4c006ee0).stop 2024-01-13T11:49:18.897 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.905+0000 7f9e6147a700 1 -- 172.21.15.59:0/886817003 shutdown_connections 2024-01-13T11:49:18.897 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.905+0000 7f9e6147a700 1 --2- 172.21.15.59:0/886817003 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f9e4802fdb0 0x7f9e48032270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:18.897 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.905+0000 7f9e6147a700 1 --2- 172.21.15.59:0/886817003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e5c0fab90 0x7f9e5c10d740 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:18.897 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.905+0000 7f9e6147a700 1 -- 172.21.15.59:0/886817003 >> 172.21.15.59:0/886817003 conn(0x7f9e5c0f63b0 msgr2=0x7f9e5c0f7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:18.898 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.905+0000 7f9e6147a700 1 -- 172.21.15.59:0/886817003 shutdown_connections 2024-01-13T11:49:18.898 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:18.905+0000 7f9e6147a700 1 -- 172.21.15.59:0/886817003 wait complete. 2024-01-13T11:49:19.268 INFO:teuthology.orchestra.run.smithi059.stderr:Deploying mgr service with default placement... 2024-01-13T11:49:19.268 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-01-13T11:49:19.634 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.640+0000 7fc0cccc1700 1 Processor -- start 2024-01-13T11:49:19.634 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.640+0000 7fc0cccc1700 1 -- start start 2024-01-13T11:49:19.635 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.641+0000 7fc0cccc1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c80703b0 0x7fc0c8100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:19.635 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.641+0000 7fc0cccc1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c80fc750 con 0x7fc0c80703b0 2024-01-13T11:49:19.635 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.641+0000 7fc0c659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c80703b0 0x7fc0c8100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:19.635 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.641+0000 7fc0c659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c80703b0 0x7fc0c8100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44392/0 (socket says 172.21.15.59:44392) 2024-01-13T11:49:19.635 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.641+0000 7fc0c659c700 1 -- 172.21.15.59:0/1337925731 learned_addr learned my addr 172.21.15.59:0/1337925731 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:19.635 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.641+0000 7fc0c659c700 1 -- 172.21.15.59:0/1337925731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c8101250 con 0x7fc0c80703b0 2024-01-13T11:49:19.635 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.641+0000 7fc0c659c700 1 --2- 172.21.15.59:0/1337925731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c80703b0 0x7fc0c8100d60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fc0b800b5f0 tx=0x7fc0b8006ee0).ready entity=mon.0 client_cookie=1fff965789d0dfea server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:19.635 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.641+0000 7fc0c559a700 1 -- 172.21.15.59:0/1337925731 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0b800d600 con 0x7fc0c80703b0 2024-01-13T11:49:19.635 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.641+0000 7fc0c559a700 1 -- 172.21.15.59:0/1337925731 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc0b800dbd0 con 0x7fc0c80703b0 2024-01-13T11:49:19.635 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.641+0000 7fc0c559a700 1 -- 172.21.15.59:0/1337925731 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0b8011c00 con 0x7fc0c80703b0 2024-01-13T11:49:19.635 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.642+0000 7fc0cccc1700 1 -- 172.21.15.59:0/1337925731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c80703b0 msgr2=0x7fc0c8100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:19.635 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.642+0000 7fc0cccc1700 1 --2- 172.21.15.59:0/1337925731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c80703b0 0x7fc0c8100d60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fc0b800b5f0 tx=0x7fc0b8006ee0).stop 2024-01-13T11:49:19.636 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.642+0000 7fc0cccc1700 1 -- 172.21.15.59:0/1337925731 shutdown_connections 2024-01-13T11:49:19.636 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.642+0000 7fc0cccc1700 1 --2- 172.21.15.59:0/1337925731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c80703b0 0x7fc0c8100d60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:19.636 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.642+0000 7fc0cccc1700 1 -- 172.21.15.59:0/1337925731 >> 172.21.15.59:0/1337925731 conn(0x7fc0c80f63c0 msgr2=0x7fc0c80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:19.636 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.642+0000 7fc0cccc1700 1 -- 172.21.15.59:0/1337925731 shutdown_connections 2024-01-13T11:49:19.636 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.642+0000 7fc0cccc1700 1 -- 172.21.15.59:0/1337925731 wait complete. 2024-01-13T11:49:19.636 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.643+0000 7fc0cccc1700 1 Processor -- start 2024-01-13T11:49:19.636 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.643+0000 7fc0cccc1700 1 -- start start 2024-01-13T11:49:19.636 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.643+0000 7fc0cccc1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c80703b0 0x7fc0c810ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:19.636 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.643+0000 7fc0cccc1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c810d1e0 con 0x7fc0c80703b0 2024-01-13T11:49:19.636 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.643+0000 7fc0c659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c80703b0 0x7fc0c810ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:19.636 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.643+0000 7fc0c659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c80703b0 0x7fc0c810ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44396/0 (socket says 172.21.15.59:44396) 2024-01-13T11:49:19.636 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.643+0000 7fc0c659c700 1 -- 172.21.15.59:0/3674775144 learned_addr learned my addr 172.21.15.59:0/3674775144 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:19.636 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.643+0000 7fc0c659c700 1 -- 172.21.15.59:0/3674775144 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b800b200 con 0x7fc0c80703b0 2024-01-13T11:49:19.636 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.643+0000 7fc0c659c700 1 --2- 172.21.15.59:0/3674775144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c80703b0 0x7fc0c810ccf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fc0b800e040 tx=0x7fc0b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:19.637 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.644+0000 7fc0beffd700 1 -- 172.21.15.59:0/3674775144 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0b800d9f0 con 0x7fc0c80703b0 2024-01-13T11:49:19.637 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.644+0000 7fc0beffd700 1 -- 172.21.15.59:0/3674775144 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc0b801a9f0 con 0x7fc0c80703b0 2024-01-13T11:49:19.637 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.644+0000 7fc0cccc1700 1 -- 172.21.15.59:0/3674775144 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c810d3e0 con 0x7fc0c80703b0 2024-01-13T11:49:19.637 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.644+0000 7fc0beffd700 1 -- 172.21.15.59:0/3674775144 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0b801b8c0 con 0x7fc0c80703b0 2024-01-13T11:49:19.637 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.644+0000 7fc0cccc1700 1 -- 172.21.15.59:0/3674775144 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c810d7f0 con 0x7fc0c80703b0 2024-01-13T11:49:19.638 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.644+0000 7fc0beffd700 1 -- 172.21.15.59:0/3674775144 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fc0b800f070 con 0x7fc0c80703b0 2024-01-13T11:49:19.638 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.645+0000 7fc0beffd700 1 --2- 172.21.15.59:0/3674775144 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7fc0a802fdb0 0x7fc0a8032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:19.638 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.645+0000 7fc0cccc1700 1 -- 172.21.15.59:0/3674775144 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c8045ad0 con 0x7fc0c80703b0 2024-01-13T11:49:19.638 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.645+0000 7fc0beffd700 1 -- 172.21.15.59:0/3674775144 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fc0b8041380 con 0x7fc0c80703b0 2024-01-13T11:49:19.638 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.645+0000 7fc0c5d9b700 1 --2- 172.21.15.59:0/3674775144 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7fc0a802fdb0 0x7fc0a8032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:19.638 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.645+0000 7fc0c5d9b700 1 --2- 172.21.15.59:0/3674775144 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7fc0a802fdb0 0x7fc0a8032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fc0b00096a0 tx=0x7fc0b0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:19.639 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.649+0000 7fc0beffd700 1 -- 172.21.15.59:0/3674775144 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0b8016070 con 0x7fc0c80703b0 2024-01-13T11:49:19.827 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.836+0000 7fc0cccc1700 1 -- 172.21.15.59:0/3674775144 --> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0c810df20 con 0x7fc0a802fdb0 2024-01-13T11:49:19.830 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.840+0000 7fc0beffd700 1 -- 172.21.15.59:0/3674775144 <== mgr.14114 v2:172.21.15.59:6800/2535313957 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc0c810df20 con 0x7fc0a802fdb0 2024-01-13T11:49:19.830 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-01-13T11:49:19.833 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.842+0000 7fc0cccc1700 1 -- 172.21.15.59:0/3674775144 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7fc0a802fdb0 msgr2=0x7fc0a8032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:19.833 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.842+0000 7fc0cccc1700 1 --2- 172.21.15.59:0/3674775144 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7fc0a802fdb0 0x7fc0a8032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fc0b00096a0 tx=0x7fc0b0006b40).stop 2024-01-13T11:49:19.833 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.842+0000 7fc0cccc1700 1 -- 172.21.15.59:0/3674775144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c80703b0 msgr2=0x7fc0c810ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:19.834 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.842+0000 7fc0cccc1700 1 --2- 172.21.15.59:0/3674775144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c80703b0 0x7fc0c810ccf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fc0b800e040 tx=0x7fc0b8006ee0).stop 2024-01-13T11:49:19.834 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.842+0000 7fc0cccc1700 1 -- 172.21.15.59:0/3674775144 shutdown_connections 2024-01-13T11:49:19.834 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.842+0000 7fc0cccc1700 1 --2- 172.21.15.59:0/3674775144 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7fc0a802fdb0 0x7fc0a8032270 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:19.834 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.842+0000 7fc0cccc1700 1 --2- 172.21.15.59:0/3674775144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c80703b0 0x7fc0c810ccf0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:19.834 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.842+0000 7fc0cccc1700 1 -- 172.21.15.59:0/3674775144 >> 172.21.15.59:0/3674775144 conn(0x7fc0c80f63c0 msgr2=0x7fc0c80f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:19.834 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.842+0000 7fc0cccc1700 1 -- 172.21.15.59:0/3674775144 shutdown_connections 2024-01-13T11:49:19.834 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:19.842+0000 7fc0cccc1700 1 -- 172.21.15.59:0/3674775144 wait complete. 2024-01-13T11:49:20.128 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:19 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:18.901319+0000 mgr.smithi059.hmyldj (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:49:20.128 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:19 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cephadm 2024-01-13T11:49:18.902079+0000 mgr.smithi059.hmyldj (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-01-13T11:49:20.128 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:19 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:18.904410+0000 mon.smithi059 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:20.128 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:19 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:19.841360+0000 mon.smithi059 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:20.192 INFO:teuthology.orchestra.run.smithi059.stderr:Deploying crash service with default placement... 2024-01-13T11:49:20.193 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-01-13T11:49:20.596 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.601+0000 7fb3ea1da700 1 Processor -- start 2024-01-13T11:49:20.596 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.602+0000 7fb3ea1da700 1 -- start start 2024-01-13T11:49:20.596 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.602+0000 7fb3ea1da700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e40703b0 0x7fb3e4100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:20.596 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.602+0000 7fb3ea1da700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e40fc750 con 0x7fb3e40703b0 2024-01-13T11:49:20.596 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.602+0000 7fb3e37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e40703b0 0x7fb3e4100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:20.596 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.602+0000 7fb3e37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e40703b0 0x7fb3e4100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44410/0 (socket says 172.21.15.59:44410) 2024-01-13T11:49:20.596 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.602+0000 7fb3e37fe700 1 -- 172.21.15.59:0/3703167680 learned_addr learned my addr 172.21.15.59:0/3703167680 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:20.596 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.602+0000 7fb3e37fe700 1 -- 172.21.15.59:0/3703167680 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e4101250 con 0x7fb3e40703b0 2024-01-13T11:49:20.596 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.602+0000 7fb3e37fe700 1 --2- 172.21.15.59:0/3703167680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e40703b0 0x7fb3e4100d60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb3d800b5f0 tx=0x7fb3d8006ee0).ready entity=mon.0 client_cookie=63492f89a67cc855 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:20.596 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.602+0000 7fb3e27fc700 1 -- 172.21.15.59:0/3703167680 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3d800d600 con 0x7fb3e40703b0 2024-01-13T11:49:20.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.603+0000 7fb3e27fc700 1 -- 172.21.15.59:0/3703167680 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb3d800dbd0 con 0x7fb3e40703b0 2024-01-13T11:49:20.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.603+0000 7fb3e27fc700 1 -- 172.21.15.59:0/3703167680 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3d8011c00 con 0x7fb3e40703b0 2024-01-13T11:49:20.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.603+0000 7fb3ea1da700 1 -- 172.21.15.59:0/3703167680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e40703b0 msgr2=0x7fb3e4100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:20.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.603+0000 7fb3ea1da700 1 --2- 172.21.15.59:0/3703167680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e40703b0 0x7fb3e4100d60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb3d800b5f0 tx=0x7fb3d8006ee0).stop 2024-01-13T11:49:20.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.603+0000 7fb3ea1da700 1 -- 172.21.15.59:0/3703167680 shutdown_connections 2024-01-13T11:49:20.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.603+0000 7fb3ea1da700 1 --2- 172.21.15.59:0/3703167680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e40703b0 0x7fb3e4100d60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:20.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.603+0000 7fb3ea1da700 1 -- 172.21.15.59:0/3703167680 >> 172.21.15.59:0/3703167680 conn(0x7fb3e40f63c0 msgr2=0x7fb3e40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:20.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.603+0000 7fb3ea1da700 1 -- 172.21.15.59:0/3703167680 shutdown_connections 2024-01-13T11:49:20.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.603+0000 7fb3ea1da700 1 -- 172.21.15.59:0/3703167680 wait complete. 2024-01-13T11:49:20.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.604+0000 7fb3ea1da700 1 Processor -- start 2024-01-13T11:49:20.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.604+0000 7fb3ea1da700 1 -- start start 2024-01-13T11:49:20.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.604+0000 7fb3ea1da700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e40703b0 0x7fb3e410ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:20.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.604+0000 7fb3ea1da700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e410d1e0 con 0x7fb3e40703b0 2024-01-13T11:49:20.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.604+0000 7fb3e37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e40703b0 0x7fb3e410ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:20.598 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.604+0000 7fb3e37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e40703b0 0x7fb3e410ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44418/0 (socket says 172.21.15.59:44418) 2024-01-13T11:49:20.598 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.604+0000 7fb3e37fe700 1 -- 172.21.15.59:0/3553127387 learned_addr learned my addr 172.21.15.59:0/3553127387 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:20.598 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.604+0000 7fb3e37fe700 1 -- 172.21.15.59:0/3553127387 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d800b200 con 0x7fb3e40703b0 2024-01-13T11:49:20.598 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.605+0000 7fb3e37fe700 1 --2- 172.21.15.59:0/3553127387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e40703b0 0x7fb3e410ccf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb3d800e040 tx=0x7fb3d8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:20.598 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.605+0000 7fb3cbfff700 1 -- 172.21.15.59:0/3553127387 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3d800d9f0 con 0x7fb3e40703b0 2024-01-13T11:49:20.598 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.605+0000 7fb3cbfff700 1 -- 172.21.15.59:0/3553127387 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb3d801a9f0 con 0x7fb3e40703b0 2024-01-13T11:49:20.598 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.605+0000 7fb3ea1da700 1 -- 172.21.15.59:0/3553127387 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e410d3e0 con 0x7fb3e40703b0 2024-01-13T11:49:20.598 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.605+0000 7fb3cbfff700 1 -- 172.21.15.59:0/3553127387 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3d801b8c0 con 0x7fb3e40703b0 2024-01-13T11:49:20.598 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.605+0000 7fb3ea1da700 1 -- 172.21.15.59:0/3553127387 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e410d7f0 con 0x7fb3e40703b0 2024-01-13T11:49:20.599 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.606+0000 7fb3cbfff700 1 -- 172.21.15.59:0/3553127387 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb3d800f070 con 0x7fb3e40703b0 2024-01-13T11:49:20.599 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.606+0000 7fb3cbfff700 1 --2- 172.21.15.59:0/3553127387 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7fb3d002fdb0 0x7fb3d0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:20.599 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.606+0000 7fb3cbfff700 1 -- 172.21.15.59:0/3553127387 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb3d80412a0 con 0x7fb3e40703b0 2024-01-13T11:49:20.599 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.606+0000 7fb3e2ffd700 1 --2- 172.21.15.59:0/3553127387 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7fb3d002fdb0 0x7fb3d0032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:20.599 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.606+0000 7fb3ea1da700 1 -- 172.21.15.59:0/3553127387 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e4101db0 con 0x7fb3e40703b0 2024-01-13T11:49:20.599 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.606+0000 7fb3e2ffd700 1 --2- 172.21.15.59:0/3553127387 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7fb3d002fdb0 0x7fb3d0032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fb3d40096a0 tx=0x7fb3d4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:20.601 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.610+0000 7fb3cbfff700 1 -- 172.21.15.59:0/3553127387 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3d8016070 con 0x7fb3e40703b0 2024-01-13T11:49:20.786 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.795+0000 7fb3ea1da700 1 -- 172.21.15.59:0/3553127387 --> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3e4101f70 con 0x7fb3d002fdb0 2024-01-13T11:49:20.790 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.799+0000 7fb3cbfff700 1 -- 172.21.15.59:0/3553127387 <== mgr.14114 v2:172.21.15.59:6800/2535313957 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fb3e4101f70 con 0x7fb3d002fdb0 2024-01-13T11:49:20.790 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: Scheduled crash update... 2024-01-13T11:49:20.792 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.801+0000 7fb3ea1da700 1 -- 172.21.15.59:0/3553127387 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7fb3d002fdb0 msgr2=0x7fb3d0032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:20.792 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.801+0000 7fb3ea1da700 1 --2- 172.21.15.59:0/3553127387 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7fb3d002fdb0 0x7fb3d0032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fb3d40096a0 tx=0x7fb3d4006b40).stop 2024-01-13T11:49:20.792 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.801+0000 7fb3ea1da700 1 -- 172.21.15.59:0/3553127387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e40703b0 msgr2=0x7fb3e410ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:20.792 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.801+0000 7fb3ea1da700 1 --2- 172.21.15.59:0/3553127387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e40703b0 0x7fb3e410ccf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb3d800e040 tx=0x7fb3d8006ee0).stop 2024-01-13T11:49:20.792 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.801+0000 7fb3ea1da700 1 -- 172.21.15.59:0/3553127387 shutdown_connections 2024-01-13T11:49:20.793 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.801+0000 7fb3ea1da700 1 --2- 172.21.15.59:0/3553127387 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7fb3d002fdb0 0x7fb3d0032270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:20.793 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.801+0000 7fb3ea1da700 1 --2- 172.21.15.59:0/3553127387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e40703b0 0x7fb3e410ccf0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:20.793 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.801+0000 7fb3ea1da700 1 -- 172.21.15.59:0/3553127387 >> 172.21.15.59:0/3553127387 conn(0x7fb3e40f63c0 msgr2=0x7fb3e40f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:20.793 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.801+0000 7fb3ea1da700 1 -- 172.21.15.59:0/3553127387 shutdown_connections 2024-01-13T11:49:20.793 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:20.801+0000 7fb3ea1da700 1 -- 172.21.15.59:0/3553127387 wait complete. 2024-01-13T11:49:21.160 INFO:teuthology.orchestra.run.smithi059.stderr:Enabling mgr prometheus module... 2024-01-13T11:49:21.160 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-01-13T11:49:21.558 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.567+0000 7f9043abe700 1 Processor -- start 2024-01-13T11:49:21.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.567+0000 7f9043abe700 1 -- start start 2024-01-13T11:49:21.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.567+0000 7f9043abe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f903c0fab80 0x7f903c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:21.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.567+0000 7f9043abe700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c0fb490 con 0x7f903c0fab80 2024-01-13T11:49:21.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.567+0000 7f904185a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f903c0fab80 0x7f903c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:21.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.567+0000 7f904185a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f903c0fab80 0x7f903c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44432/0 (socket says 172.21.15.59:44432) 2024-01-13T11:49:21.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.567+0000 7f904185a700 1 -- 172.21.15.59:0/3874005233 learned_addr learned my addr 172.21.15.59:0/3874005233 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:21.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.568+0000 7f904185a700 1 -- 172.21.15.59:0/3874005233 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903c0fb5d0 con 0x7f903c0fab80 2024-01-13T11:49:21.565 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.568+0000 7f904185a700 1 --2- 172.21.15.59:0/3874005233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f903c0fab80 0x7f903c0fafa0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f903800b5f0 tx=0x7f9038006ee0).ready entity=mon.0 client_cookie=72287b9f36f7d64d server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:21.565 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.568+0000 7f9040858700 1 -- 172.21.15.59:0/3874005233 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f903800d600 con 0x7f903c0fab80 2024-01-13T11:49:21.565 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.568+0000 7f9040858700 1 -- 172.21.15.59:0/3874005233 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f903800dbd0 con 0x7f903c0fab80 2024-01-13T11:49:21.565 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.568+0000 7f9040858700 1 -- 172.21.15.59:0/3874005233 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9038011c00 con 0x7f903c0fab80 2024-01-13T11:49:21.565 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.569+0000 7f9043abe700 1 -- 172.21.15.59:0/3874005233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f903c0fab80 msgr2=0x7f903c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:21.565 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.569+0000 7f9043abe700 1 --2- 172.21.15.59:0/3874005233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f903c0fab80 0x7f903c0fafa0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f903800b5f0 tx=0x7f9038006ee0).stop 2024-01-13T11:49:21.565 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.569+0000 7f9043abe700 1 -- 172.21.15.59:0/3874005233 shutdown_connections 2024-01-13T11:49:21.565 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.569+0000 7f9043abe700 1 --2- 172.21.15.59:0/3874005233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f903c0fab80 0x7f903c0fafa0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:21.565 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.569+0000 7f9043abe700 1 -- 172.21.15.59:0/3874005233 >> 172.21.15.59:0/3874005233 conn(0x7f903c0f63c0 msgr2=0x7f903c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.569+0000 7f9043abe700 1 -- 172.21.15.59:0/3874005233 shutdown_connections 2024-01-13T11:49:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.569+0000 7f9043abe700 1 -- 172.21.15.59:0/3874005233 wait complete. 2024-01-13T11:49:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.570+0000 7f9043abe700 1 Processor -- start 2024-01-13T11:49:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.570+0000 7f9043abe700 1 -- start start 2024-01-13T11:49:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.570+0000 7f9043abe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f903c0fab80 0x7f903c10d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.570+0000 7f9043abe700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c070470 con 0x7f903c0fab80 2024-01-13T11:49:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.570+0000 7f904185a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f903c0fab80 0x7f903c10d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.570+0000 7f904185a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f903c0fab80 0x7f903c10d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44446/0 (socket says 172.21.15.59:44446) 2024-01-13T11:49:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.570+0000 7f904185a700 1 -- 172.21.15.59:0/2671015436 learned_addr learned my addr 172.21.15.59:0/2671015436 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.570+0000 7f904185a700 1 -- 172.21.15.59:0/2671015436 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903800b200 con 0x7f903c0fab80 2024-01-13T11:49:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.570+0000 7f904185a700 1 --2- 172.21.15.59:0/2671015436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f903c0fab80 0x7f903c10d6e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f903800b1d0 tx=0x7f9038006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.570+0000 7f902a7fc700 1 -- 172.21.15.59:0/2671015436 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f903801b410 con 0x7f903c0fab80 2024-01-13T11:49:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.570+0000 7f902a7fc700 1 -- 172.21.15.59:0/2671015436 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f903801b9e0 con 0x7f903c0fab80 2024-01-13T11:49:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.571+0000 7f902a7fc700 1 -- 172.21.15.59:0/2671015436 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f903801a840 con 0x7f903c0fab80 2024-01-13T11:49:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.571+0000 7f9043abe700 1 -- 172.21.15.59:0/2671015436 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903c070670 con 0x7f903c0fab80 2024-01-13T11:49:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.571+0000 7f9043abe700 1 -- 172.21.15.59:0/2671015436 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f903c10dd90 con 0x7f903c0fab80 2024-01-13T11:49:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.571+0000 7f9043abe700 1 -- 172.21.15.59:0/2671015436 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f903c045ad0 con 0x7f903c0fab80 2024-01-13T11:49:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.571+0000 7f902a7fc700 1 -- 172.21.15.59:0/2671015436 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f903800f070 con 0x7f903c0fab80 2024-01-13T11:49:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.572+0000 7f902a7fc700 1 --2- 172.21.15.59:0/2671015436 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f902402fd50 0x7f9024032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.572+0000 7f902a7fc700 1 -- 172.21.15.59:0/2671015436 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f9038041240 con 0x7f903c0fab80 2024-01-13T11:49:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.572+0000 7f9041059700 1 --2- 172.21.15.59:0/2671015436 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f902402fd50 0x7f9024032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.572+0000 7f9041059700 1 --2- 172.21.15.59:0/2671015436 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f902402fd50 0x7f9024032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f902c0096a0 tx=0x7f902c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:21.568 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.576+0000 7f902a7fc700 1 -- 172.21.15.59:0/2671015436 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9038019c90 con 0x7f903c0fab80 2024-01-13T11:49:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:21.774+0000 7f9043abe700 1 -- 172.21.15.59:0/2671015436 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f903c0fc890 con 0x7f903c0fab80 2024-01-13T11:49:22.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:21 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:19.838718+0000 mgr.smithi059.hmyldj (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:49:22.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:21 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cephadm 2024-01-13T11:49:19.839243+0000 mgr.smithi059.hmyldj (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-01-13T11:49:22.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:21 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:20.797720+0000 mgr.smithi059.hmyldj (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:49:22.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:21 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cephadm 2024-01-13T11:49:20.798338+0000 mgr.smithi059.hmyldj (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-01-13T11:49:22.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:21 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:20.800655+0000 mon.smithi059 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.59:0/654175293' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:22.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:21 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:21.776497+0000 mon.smithi059 (mon.0) 59 : audit [INF] from='client.? 172.21.15.59:0/2671015436' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-13T11:49:22.170 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.179+0000 7f902a7fc700 1 -- 172.21.15.59:0/2671015436 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f903801aab0 con 0x7f903c0fab80 2024-01-13T11:49:22.171 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.180+0000 7f902a7fc700 1 -- 172.21.15.59:0/2671015436 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f90380243c0 con 0x7f903c0fab80 2024-01-13T11:49:22.174 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.183+0000 7f9043abe700 1 -- 172.21.15.59:0/2671015436 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f902402fd50 msgr2=0x7f9024032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:22.174 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.183+0000 7f9043abe700 1 --2- 172.21.15.59:0/2671015436 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f902402fd50 0x7f9024032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f902c0096a0 tx=0x7f902c006b40).stop 2024-01-13T11:49:22.174 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.183+0000 7f9043abe700 1 -- 172.21.15.59:0/2671015436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f903c0fab80 msgr2=0x7f903c10d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:22.175 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.183+0000 7f9043abe700 1 --2- 172.21.15.59:0/2671015436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f903c0fab80 0x7f903c10d6e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f903800b1d0 tx=0x7f9038006ee0).stop 2024-01-13T11:49:22.175 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.183+0000 7f9043abe700 1 -- 172.21.15.59:0/2671015436 shutdown_connections 2024-01-13T11:49:22.175 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.183+0000 7f9043abe700 1 --2- 172.21.15.59:0/2671015436 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7f902402fd50 0x7f9024032210 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:22.175 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.183+0000 7f9043abe700 1 --2- 172.21.15.59:0/2671015436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f903c0fab80 0x7f903c10d6e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:22.175 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.183+0000 7f9043abe700 1 -- 172.21.15.59:0/2671015436 >> 172.21.15.59:0/2671015436 conn(0x7f903c0f63c0 msgr2=0x7f903c0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:22.175 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.183+0000 7f9043abe700 1 -- 172.21.15.59:0/2671015436 shutdown_connections 2024-01-13T11:49:22.176 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.183+0000 7f9043abe700 1 -- 172.21.15.59:0/2671015436 wait complete. 2024-01-13T11:49:22.534 INFO:teuthology.orchestra.run.smithi059.stderr:Deploying prometheus service with default placement... 2024-01-13T11:49:22.534 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-01-13T11:49:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.971+0000 7ff2e5515700 1 Processor -- start 2024-01-13T11:49:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.971+0000 7ff2e5515700 1 -- start start 2024-01-13T11:49:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.971+0000 7ff2e5515700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00703b0 0x7ff2e0100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.971+0000 7ff2e5515700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e00fc5a0 con 0x7ff2e00703b0 2024-01-13T11:49:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.971+0000 7ff2dffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00703b0 0x7ff2e0100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.971+0000 7ff2dffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00703b0 0x7ff2e0100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44468/0 (socket says 172.21.15.59:44468) 2024-01-13T11:49:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.971+0000 7ff2dffff700 1 -- 172.21.15.59:0/2551414585 learned_addr learned my addr 172.21.15.59:0/2551414585 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.972+0000 7ff2dffff700 1 -- 172.21.15.59:0/2551414585 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e01010a0 con 0x7ff2e00703b0 2024-01-13T11:49:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.972+0000 7ff2dffff700 1 --2- 172.21.15.59:0/2551414585 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00703b0 0x7ff2e0100bb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7ff2d000b5f0 tx=0x7ff2d0006ee0).ready entity=mon.0 client_cookie=27f0a80fb1912f3f server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.972+0000 7ff2deffd700 1 -- 172.21.15.59:0/2551414585 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2d000d600 con 0x7ff2e00703b0 2024-01-13T11:49:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.972+0000 7ff2deffd700 1 -- 172.21.15.59:0/2551414585 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff2d000dbd0 con 0x7ff2e00703b0 2024-01-13T11:49:22.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.972+0000 7ff2deffd700 1 -- 172.21.15.59:0/2551414585 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2d0011c00 con 0x7ff2e00703b0 2024-01-13T11:49:22.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.972+0000 7ff2e5515700 1 -- 172.21.15.59:0/2551414585 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00703b0 msgr2=0x7ff2e0100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:22.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.972+0000 7ff2e5515700 1 --2- 172.21.15.59:0/2551414585 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00703b0 0x7ff2e0100bb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7ff2d000b5f0 tx=0x7ff2d0006ee0).stop 2024-01-13T11:49:22.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.973+0000 7ff2e5515700 1 -- 172.21.15.59:0/2551414585 shutdown_connections 2024-01-13T11:49:22.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.973+0000 7ff2e5515700 1 --2- 172.21.15.59:0/2551414585 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00703b0 0x7ff2e0100bb0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:22.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.973+0000 7ff2e5515700 1 -- 172.21.15.59:0/2551414585 >> 172.21.15.59:0/2551414585 conn(0x7ff2e00f6210 msgr2=0x7ff2e00f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:22.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.973+0000 7ff2e5515700 1 -- 172.21.15.59:0/2551414585 shutdown_connections 2024-01-13T11:49:22.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.973+0000 7ff2e5515700 1 -- 172.21.15.59:0/2551414585 wait complete. 2024-01-13T11:49:22.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.973+0000 7ff2e5515700 1 Processor -- start 2024-01-13T11:49:22.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.973+0000 7ff2e5515700 1 -- start start 2024-01-13T11:49:22.967 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.974+0000 7ff2e5515700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00703b0 0x7ff2e010ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:22.967 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.974+0000 7ff2e5515700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e010cf60 con 0x7ff2e00703b0 2024-01-13T11:49:22.967 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.974+0000 7ff2dffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00703b0 0x7ff2e010ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:22.967 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.974+0000 7ff2dffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00703b0 0x7ff2e010ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44476/0 (socket says 172.21.15.59:44476) 2024-01-13T11:49:22.967 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.974+0000 7ff2dffff700 1 -- 172.21.15.59:0/4238182418 learned_addr learned my addr 172.21.15.59:0/4238182418 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:22.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.974+0000 7ff2dffff700 1 -- 172.21.15.59:0/4238182418 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d000b200 con 0x7ff2e00703b0 2024-01-13T11:49:22.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.974+0000 7ff2dffff700 1 --2- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00703b0 0x7ff2e010ca70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7ff2d000e040 tx=0x7ff2d0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:22.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.974+0000 7ff2dcff9700 1 -- 172.21.15.59:0/4238182418 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2d000d9f0 con 0x7ff2e00703b0 2024-01-13T11:49:22.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.974+0000 7ff2dcff9700 1 -- 172.21.15.59:0/4238182418 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff2d001a9f0 con 0x7ff2e00703b0 2024-01-13T11:49:22.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.974+0000 7ff2e5515700 1 -- 172.21.15.59:0/4238182418 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e010d160 con 0x7ff2e00703b0 2024-01-13T11:49:22.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.974+0000 7ff2dcff9700 1 -- 172.21.15.59:0/4238182418 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2d001b8c0 con 0x7ff2e00703b0 2024-01-13T11:49:22.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.974+0000 7ff2e5515700 1 -- 172.21.15.59:0/4238182418 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e010d570 con 0x7ff2e00703b0 2024-01-13T11:49:22.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.975+0000 7ff2dcff9700 1 -- 172.21.15.59:0/4238182418 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7ff2d000f070 con 0x7ff2e00703b0 2024-01-13T11:49:22.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.976+0000 7ff2dcff9700 1 --2- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7ff2cc02fdb0 0x7ff2cc032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:22.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.976+0000 7ff2dcff9700 1 -- 172.21.15.59:0/4238182418 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff2d0041330 con 0x7ff2e00703b0 2024-01-13T11:49:22.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.976+0000 7ff2df7fe700 1 -- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7ff2cc02fdb0 msgr2=0x7ff2cc032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/2535313957 2024-01-13T11:49:22.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.976+0000 7ff2df7fe700 1 --2- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7ff2cc02fdb0 0x7ff2cc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-13T11:49:22.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.976+0000 7ff2e5515700 1 -- 172.21.15.59:0/4238182418 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e0059100 con 0x7ff2e00703b0 2024-01-13T11:49:22.970 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:22.980+0000 7ff2dcff9700 1 -- 172.21.15.59:0/4238182418 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2d0016070 con 0x7ff2e00703b0 2024-01-13T11:49:23.161 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:23.170+0000 7ff2e5515700 1 -- 172.21.15.59:0/4238182418 --> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2e0106950 con 0x7ff2cc02fdb0 2024-01-13T11:49:23.167 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:23.177+0000 7ff2df7fe700 1 -- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7ff2cc02fdb0 msgr2=0x7ff2cc032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/2535313957 2024-01-13T11:49:23.167 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:23.177+0000 7ff2df7fe700 1 --2- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7ff2cc02fdb0 0x7ff2cc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-13T11:49:23.568 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:23.578+0000 7ff2df7fe700 1 -- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7ff2cc02fdb0 msgr2=0x7ff2cc032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/2535313957 2024-01-13T11:49:23.569 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:23.578+0000 7ff2df7fe700 1 --2- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7ff2cc02fdb0 0x7ff2cc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-13T11:49:23.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:23 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:22.182274+0000 mon.smithi059 (mon.0) 60 : audit [INF] from='client.? 172.21.15.59:0/2671015436' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-13T11:49:23.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:23 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:22.182330+0000 mon.smithi059 (mon.0) 61 : cluster [DBG] mgrmap e9: smithi059.hmyldj(active, since 16s) 2024-01-13T11:49:24.369 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:24.379+0000 7ff2df7fe700 1 -- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7ff2cc02fdb0 msgr2=0x7ff2cc032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/2535313957 2024-01-13T11:49:24.370 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:24.379+0000 7ff2df7fe700 1 --2- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7ff2cc02fdb0 0x7ff2cc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-13T11:49:25.971 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:25.980+0000 7ff2df7fe700 1 -- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7ff2cc02fdb0 msgr2=0x7ff2cc032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/2535313957 2024-01-13T11:49:25.971 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:25.980+0000 7ff2df7fe700 1 --2- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7ff2cc02fdb0 0x7ff2cc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-13T11:49:27.372 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:27.381+0000 7ff2dcff9700 1 -- 172.21.15.59:0/4238182418 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7ff2d00257e0 con 0x7ff2e00703b0 2024-01-13T11:49:27.373 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:27.381+0000 7ff2dcff9700 1 -- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7ff2cc02fdb0 msgr2=0x7ff2cc032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-13T11:49:27.373 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:27.382+0000 7ff2dcff9700 1 --2- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7ff2cc02fdb0 0x7ff2cc032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-13T11:49:27.627 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:27 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:27.331206+0000 mon.smithi059 (mon.0) 62 : cluster [INF] Active manager daemon smithi059.hmyldj restarted 2024-01-13T11:49:27.627 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:27 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:27.331607+0000 mon.smithi059 (mon.0) 63 : cluster [INF] Activating manager daemon smithi059.hmyldj 2024-01-13T11:49:27.627 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:27 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:27.333254+0000 mon.smithi059 (mon.0) 64 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-01-13T11:49:28.383 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:28.390+0000 7ff2dcff9700 1 -- 172.21.15.59:0/4238182418 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7ff2d00433c0 con 0x7ff2e00703b0 2024-01-13T11:49:28.383 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:28.390+0000 7ff2dcff9700 1 --2- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7ff2cc033c20 0x7ff2cc036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:28.383 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:28.390+0000 7ff2dcff9700 1 -- 172.21.15.59:0/4238182418 --> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2e0106950 con 0x7ff2cc033c20 2024-01-13T11:49:28.383 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:28.391+0000 7ff2df7fe700 1 --2- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7ff2cc033c20 0x7ff2cc036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:28.383 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:28.391+0000 7ff2df7fe700 1 --2- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7ff2cc033c20 0x7ff2cc036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ff2d40036d0 tx=0x7ff2d4006cb0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:28.386 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:28.396+0000 7ff2dcff9700 1 -- 172.21.15.59:0/4238182418 <== mgr.14142 v2:172.21.15.59:6800/373398304 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7ff2e0106950 con 0x7ff2cc033c20 2024-01-13T11:49:28.387 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-01-13T11:49:28.390 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:28.398+0000 7ff2e5515700 1 -- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7ff2cc033c20 msgr2=0x7ff2cc036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:28.390 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:28.398+0000 7ff2e5515700 1 --2- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7ff2cc033c20 0x7ff2cc036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ff2d40036d0 tx=0x7ff2d4006cb0).stop 2024-01-13T11:49:28.390 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:28.398+0000 7ff2e5515700 1 -- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00703b0 msgr2=0x7ff2e010ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:28.390 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:28.398+0000 7ff2e5515700 1 --2- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00703b0 0x7ff2e010ca70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7ff2d000e040 tx=0x7ff2d0006ee0).stop 2024-01-13T11:49:28.390 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:28.398+0000 7ff2e5515700 1 -- 172.21.15.59:0/4238182418 shutdown_connections 2024-01-13T11:49:28.390 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:28.398+0000 7ff2e5515700 1 --2- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7ff2cc033c20 0x7ff2cc036010 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:28.390 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:28.398+0000 7ff2e5515700 1 --2- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00703b0 0x7ff2e010ca70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:28.390 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:28.398+0000 7ff2e5515700 1 --2- 172.21.15.59:0/4238182418 >> [v2:172.21.15.59:6800/2535313957,v1:172.21.15.59:6801/2535313957] conn(0x7ff2cc02fdb0 0x7ff2cc032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-13T11:49:28.390 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:28.398+0000 7ff2e5515700 1 -- 172.21.15.59:0/4238182418 >> 172.21.15.59:0/4238182418 conn(0x7ff2e00f6210 msgr2=0x7ff2e00f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:28.390 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:28.398+0000 7ff2e5515700 1 -- 172.21.15.59:0/4238182418 shutdown_connections 2024-01-13T11:49:28.390 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:28.398+0000 7ff2e5515700 1 -- 172.21.15.59:0/4238182418 wait complete. 2024-01-13T11:49:28.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:28 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:27.384369+0000 mon.smithi059 (mon.0) 65 : cluster [DBG] mgrmap e10: smithi059.hmyldj(active, starting, since 0.052842s) 2024-01-13T11:49:28.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:28 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:27.385381+0000 mon.smithi059 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.59:0/2761322198' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-01-13T11:49:28.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:28 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:27.387565+0000 mon.smithi059 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.59:0/2761322198' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "mgr metadata", "who": "smithi059.hmyldj", "id": "smithi059.hmyldj"}]: dispatch 2024-01-13T11:49:28.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:28 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:27.387765+0000 mon.smithi059 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.59:0/2761322198' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T11:49:28.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:28 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:27.387887+0000 mon.smithi059 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.59:0/2761322198' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T11:49:28.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:28 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:27.388013+0000 mon.smithi059 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.59:0/2761322198' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T11:49:28.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:28 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:27.401883+0000 mon.smithi059 (mon.0) 71 : cluster [INF] Manager daemon smithi059.hmyldj is now available 2024-01-13T11:49:28.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:28 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:27.504866+0000 mon.smithi059 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.59:0/2761322198' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:49:28.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:28 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:27.506324+0000 mon.smithi059 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.59:0/2761322198' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:49:28.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:28 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:27.526798+0000 mon.smithi059 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.59:0/2761322198' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/mirror_snapshot_schedule"}]: dispatch 2024-01-13T11:49:28.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:28 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:27.529311+0000 mon.smithi059 (mon.0) 75 : audit [INF] from='mgr.14142 172.21.15.59:0/2761322198' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/trash_purge_schedule"}]: dispatch 2024-01-13T11:49:28.733 INFO:teuthology.orchestra.run.smithi059.stderr:Deploying grafana service with default placement... 2024-01-13T11:49:28.733 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-01-13T11:49:29.182 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.187+0000 7fca121aa700 1 Processor -- start 2024-01-13T11:49:29.182 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.187+0000 7fca121aa700 1 -- start start 2024-01-13T11:49:29.183 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.187+0000 7fca121aa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0703d0 0x7fca0c100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:29.183 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.187+0000 7fca121aa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c0fc730 con 0x7fca0c0703d0 2024-01-13T11:49:29.183 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.187+0000 7fca0b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0703d0 0x7fca0c100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:29.183 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.187+0000 7fca0b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0703d0 0x7fca0c100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44532/0 (socket says 172.21.15.59:44532) 2024-01-13T11:49:29.183 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.187+0000 7fca0b7fe700 1 -- 172.21.15.59:0/4124220774 learned_addr learned my addr 172.21.15.59:0/4124220774 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:29.183 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.188+0000 7fca0b7fe700 1 -- 172.21.15.59:0/4124220774 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c101230 con 0x7fca0c0703d0 2024-01-13T11:49:29.183 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.188+0000 7fca0b7fe700 1 --2- 172.21.15.59:0/4124220774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0703d0 0x7fca0c100d40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fca0000b5f0 tx=0x7fca00006ee0).ready entity=mon.0 client_cookie=ae3ebf13f9ed37b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:29.183 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.188+0000 7fca0a7fc700 1 -- 172.21.15.59:0/4124220774 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca0000d600 con 0x7fca0c0703d0 2024-01-13T11:49:29.183 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.188+0000 7fca0a7fc700 1 -- 172.21.15.59:0/4124220774 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fca0000dbd0 con 0x7fca0c0703d0 2024-01-13T11:49:29.183 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.188+0000 7fca0a7fc700 1 -- 172.21.15.59:0/4124220774 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca00011c00 con 0x7fca0c0703d0 2024-01-13T11:49:29.183 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.188+0000 7fca121aa700 1 -- 172.21.15.59:0/4124220774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0703d0 msgr2=0x7fca0c100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:29.183 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.188+0000 7fca121aa700 1 --2- 172.21.15.59:0/4124220774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0703d0 0x7fca0c100d40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fca0000b5f0 tx=0x7fca00006ee0).stop 2024-01-13T11:49:29.183 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.188+0000 7fca121aa700 1 -- 172.21.15.59:0/4124220774 shutdown_connections 2024-01-13T11:49:29.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.188+0000 7fca121aa700 1 --2- 172.21.15.59:0/4124220774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0703d0 0x7fca0c100d40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:29.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.188+0000 7fca121aa700 1 -- 172.21.15.59:0/4124220774 >> 172.21.15.59:0/4124220774 conn(0x7fca0c0f63c0 msgr2=0x7fca0c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:29.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.189+0000 7fca121aa700 1 -- 172.21.15.59:0/4124220774 shutdown_connections 2024-01-13T11:49:29.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.189+0000 7fca121aa700 1 -- 172.21.15.59:0/4124220774 wait complete. 2024-01-13T11:49:29.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.189+0000 7fca121aa700 1 Processor -- start 2024-01-13T11:49:29.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.189+0000 7fca121aa700 1 -- start start 2024-01-13T11:49:29.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.190+0000 7fca121aa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0703d0 0x7fca0c10cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:29.184 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.190+0000 7fca121aa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c10d0a0 con 0x7fca0c0703d0 2024-01-13T11:49:29.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.190+0000 7fca0b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0703d0 0x7fca0c10cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:29.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.190+0000 7fca0b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0703d0 0x7fca0c10cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44538/0 (socket says 172.21.15.59:44538) 2024-01-13T11:49:29.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.190+0000 7fca0b7fe700 1 -- 172.21.15.59:0/923548479 learned_addr learned my addr 172.21.15.59:0/923548479 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:29.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.190+0000 7fca0b7fe700 1 -- 172.21.15.59:0/923548479 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0000b200 con 0x7fca0c0703d0 2024-01-13T11:49:29.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.190+0000 7fca0b7fe700 1 --2- 172.21.15.59:0/923548479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0703d0 0x7fca0c10cbb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fca0000b190 tx=0x7fca00006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:29.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.190+0000 7fc9f3fff700 1 -- 172.21.15.59:0/923548479 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca0001b410 con 0x7fca0c0703d0 2024-01-13T11:49:29.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.190+0000 7fc9f3fff700 1 -- 172.21.15.59:0/923548479 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fca0001b9e0 con 0x7fca0c0703d0 2024-01-13T11:49:29.185 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.190+0000 7fca121aa700 1 -- 172.21.15.59:0/923548479 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0c10d2a0 con 0x7fca0c0703d0 2024-01-13T11:49:29.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.190+0000 7fc9f3fff700 1 -- 172.21.15.59:0/923548479 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca0001a880 con 0x7fca0c0703d0 2024-01-13T11:49:29.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.190+0000 7fca121aa700 1 -- 172.21.15.59:0/923548479 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0c10d6b0 con 0x7fca0c0703d0 2024-01-13T11:49:29.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.191+0000 7fc9f3fff700 1 -- 172.21.15.59:0/923548479 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fca0000f070 con 0x7fca0c0703d0 2024-01-13T11:49:29.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.191+0000 7fc9f3fff700 1 --2- 172.21.15.59:0/923548479 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fc9f802fd50 0x7fc9f8032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:29.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.191+0000 7fc9f3fff700 1 -- 172.21.15.59:0/923548479 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fca00041200 con 0x7fca0c0703d0 2024-01-13T11:49:29.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.191+0000 7fca0affd700 1 --2- 172.21.15.59:0/923548479 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fc9f802fd50 0x7fc9f8032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:29.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.192+0000 7fca121aa700 1 -- 172.21.15.59:0/923548479 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca0c059100 con 0x7fca0c0703d0 2024-01-13T11:49:29.186 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.192+0000 7fca0affd700 1 --2- 172.21.15.59:0/923548479 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fc9f802fd50 0x7fc9f8032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc9fc0096a0 tx=0x7fc9fc006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:29.187 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.196+0000 7fc9f3fff700 1 -- 172.21.15.59:0/923548479 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca00016070 con 0x7fca0c0703d0 2024-01-13T11:49:29.374 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.383+0000 7fca121aa700 1 -- 172.21.15.59:0/923548479 --> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fca0c106a60 con 0x7fc9f802fd50 2024-01-13T11:49:29.378 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.388+0000 7fc9f3fff700 1 -- 172.21.15.59:0/923548479 <== mgr.14142 v2:172.21.15.59:6800/373398304 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fca0c106a60 con 0x7fc9f802fd50 2024-01-13T11:49:29.379 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-01-13T11:49:29.382 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.389+0000 7fca121aa700 1 -- 172.21.15.59:0/923548479 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fc9f802fd50 msgr2=0x7fc9f8032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:29.382 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.389+0000 7fca121aa700 1 --2- 172.21.15.59:0/923548479 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fc9f802fd50 0x7fc9f8032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc9fc0096a0 tx=0x7fc9fc006b40).stop 2024-01-13T11:49:29.382 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.389+0000 7fca121aa700 1 -- 172.21.15.59:0/923548479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0703d0 msgr2=0x7fca0c10cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:29.382 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.389+0000 7fca121aa700 1 --2- 172.21.15.59:0/923548479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0703d0 0x7fca0c10cbb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fca0000b190 tx=0x7fca00006ee0).stop 2024-01-13T11:49:29.382 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.389+0000 7fca121aa700 1 -- 172.21.15.59:0/923548479 shutdown_connections 2024-01-13T11:49:29.383 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.390+0000 7fca121aa700 1 --2- 172.21.15.59:0/923548479 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fc9f802fd50 0x7fc9f8032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:29.383 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.390+0000 7fca121aa700 1 --2- 172.21.15.59:0/923548479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0703d0 0x7fca0c10cbb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:29.383 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.390+0000 7fca121aa700 1 -- 172.21.15.59:0/923548479 >> 172.21.15.59:0/923548479 conn(0x7fca0c0f63c0 msgr2=0x7fca0c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:29.383 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.390+0000 7fca121aa700 1 -- 172.21.15.59:0/923548479 shutdown_connections 2024-01-13T11:49:29.383 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:29.390+0000 7fca121aa700 1 -- 172.21.15.59:0/923548479 wait complete. 2024-01-13T11:49:29.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:29 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:28.393846+0000 mon.smithi059 (mon.0) 76 : cluster [DBG] mgrmap e11: smithi059.hmyldj(active, since 1.06232s) 2024-01-13T11:49:29.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:29 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cephadm 2024-01-13T11:49:28.395145+0000 mgr.smithi059.hmyldj (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-01-13T11:49:29.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:29 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:28.397486+0000 mon.smithi059 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.59:0/2761322198' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:29.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:29 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:29.389074+0000 mon.smithi059 (mon.0) 78 : audit [INF] from='mgr.14142 172.21.15.59:0/2761322198' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:29.741 INFO:teuthology.orchestra.run.smithi059.stderr:Deploying node-exporter service with default placement... 2024-01-13T11:49:29.742 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-01-13T11:49:30.149 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.155+0000 7f4e16153700 1 Processor -- start 2024-01-13T11:49:30.149 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.155+0000 7f4e16153700 1 -- start start 2024-01-13T11:49:30.149 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.156+0000 7f4e16153700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e100fab80 0x7f4e100fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:30.149 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.156+0000 7f4e16153700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e100fb490 con 0x7f4e100fab80 2024-01-13T11:49:30.150 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.156+0000 7f4e0f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e100fab80 0x7f4e100fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:30.150 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.156+0000 7f4e0f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e100fab80 0x7f4e100fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60098/0 (socket says 172.21.15.59:60098) 2024-01-13T11:49:30.150 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.156+0000 7f4e0f7fe700 1 -- 172.21.15.59:0/3673142511 learned_addr learned my addr 172.21.15.59:0/3673142511 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:30.150 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.156+0000 7f4e0f7fe700 1 -- 172.21.15.59:0/3673142511 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e100fb5d0 con 0x7f4e100fab80 2024-01-13T11:49:30.150 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.156+0000 7f4e0f7fe700 1 --2- 172.21.15.59:0/3673142511 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e100fab80 0x7f4e100fafa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f4e0000b870 tx=0x7f4e00006f70).ready entity=mon.0 client_cookie=20d9f345867d8796 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:30.150 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.156+0000 7f4e0e7fc700 1 -- 172.21.15.59:0/3673142511 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e0000da20 con 0x7f4e100fab80 2024-01-13T11:49:30.150 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.156+0000 7f4e0e7fc700 1 -- 172.21.15.59:0/3673142511 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4e00010470 con 0x7f4e100fab80 2024-01-13T11:49:30.150 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.156+0000 7f4e0e7fc700 1 -- 172.21.15.59:0/3673142511 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e0000f5b0 con 0x7f4e100fab80 2024-01-13T11:49:30.150 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.157+0000 7f4e16153700 1 -- 172.21.15.59:0/3673142511 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e100fab80 msgr2=0x7f4e100fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:30.150 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.157+0000 7f4e16153700 1 --2- 172.21.15.59:0/3673142511 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e100fab80 0x7f4e100fafa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f4e0000b870 tx=0x7f4e00006f70).stop 2024-01-13T11:49:30.150 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.157+0000 7f4e16153700 1 -- 172.21.15.59:0/3673142511 shutdown_connections 2024-01-13T11:49:30.150 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.157+0000 7f4e16153700 1 --2- 172.21.15.59:0/3673142511 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e100fab80 0x7f4e100fafa0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:30.150 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.157+0000 7f4e16153700 1 -- 172.21.15.59:0/3673142511 >> 172.21.15.59:0/3673142511 conn(0x7f4e100f63c0 msgr2=0x7f4e100f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:30.151 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.157+0000 7f4e16153700 1 -- 172.21.15.59:0/3673142511 shutdown_connections 2024-01-13T11:49:30.151 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.157+0000 7f4e16153700 1 -- 172.21.15.59:0/3673142511 wait complete. 2024-01-13T11:49:30.151 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.158+0000 7f4e16153700 1 Processor -- start 2024-01-13T11:49:30.151 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.158+0000 7f4e16153700 1 -- start start 2024-01-13T11:49:30.151 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.158+0000 7f4e16153700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e100fab80 0x7f4e10108770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:30.151 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.158+0000 7f4e16153700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e10108c60 con 0x7f4e100fab80 2024-01-13T11:49:30.151 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.158+0000 7f4e0f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e100fab80 0x7f4e10108770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:30.151 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.158+0000 7f4e0f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e100fab80 0x7f4e10108770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60102/0 (socket says 172.21.15.59:60102) 2024-01-13T11:49:30.151 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.159+0000 7f4e0f7fe700 1 -- 172.21.15.59:0/484498982 learned_addr learned my addr 172.21.15.59:0/484498982 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:30.153 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.159+0000 7f4e0f7fe700 1 -- 172.21.15.59:0/484498982 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0000b480 con 0x7f4e100fab80 2024-01-13T11:49:30.153 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.159+0000 7f4e0f7fe700 1 --2- 172.21.15.59:0/484498982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e100fab80 0x7f4e10108770 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f4e00006a40 tx=0x7f4e00006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:30.153 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.159+0000 7f4df7fff700 1 -- 172.21.15.59:0/484498982 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e0000f3d0 con 0x7f4e100fab80 2024-01-13T11:49:30.153 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.159+0000 7f4df7fff700 1 -- 172.21.15.59:0/484498982 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4e0000fef0 con 0x7f4e100fab80 2024-01-13T11:49:30.153 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.159+0000 7f4e16153700 1 -- 172.21.15.59:0/484498982 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e10108e60 con 0x7f4e100fab80 2024-01-13T11:49:30.153 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.159+0000 7f4df7fff700 1 -- 172.21.15.59:0/484498982 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e00019d40 con 0x7f4e100fab80 2024-01-13T11:49:30.153 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.159+0000 7f4e16153700 1 -- 172.21.15.59:0/484498982 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e10109270 con 0x7f4e100fab80 2024-01-13T11:49:30.153 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.160+0000 7f4df7fff700 1 -- 172.21.15.59:0/484498982 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f4e0002b060 con 0x7f4e100fab80 2024-01-13T11:49:30.153 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.160+0000 7f4e16153700 1 -- 172.21.15.59:0/484498982 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e10045ad0 con 0x7f4e100fab80 2024-01-13T11:49:30.153 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.160+0000 7f4df7fff700 1 --2- 172.21.15.59:0/484498982 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f4dfc02fe60 0x7f4dfc032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:30.153 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.160+0000 7f4df7fff700 1 -- 172.21.15.59:0/484498982 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f4e00044250 con 0x7f4e100fab80 2024-01-13T11:49:30.153 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.160+0000 7f4e0effd700 1 --2- 172.21.15.59:0/484498982 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f4dfc02fe60 0x7f4dfc032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:30.154 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.161+0000 7f4e0effd700 1 --2- 172.21.15.59:0/484498982 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f4dfc02fe60 0x7f4dfc032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f4e040096a0 tx=0x7f4e04006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:30.155 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.165+0000 7f4df7fff700 1 -- 172.21.15.59:0/484498982 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e0002c520 con 0x7f4e100fab80 2024-01-13T11:49:30.341 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.350+0000 7f4e16153700 1 -- 172.21.15.59:0/484498982 --> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e10109bb0 con 0x7f4dfc02fe60 2024-01-13T11:49:30.345 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.354+0000 7f4df7fff700 1 -- 172.21.15.59:0/484498982 <== mgr.14142 v2:172.21.15.59:6800/373398304 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f4e10109bb0 con 0x7f4dfc02fe60 2024-01-13T11:49:30.345 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-01-13T11:49:30.347 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.355+0000 7f4e16153700 1 -- 172.21.15.59:0/484498982 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f4dfc02fe60 msgr2=0x7f4dfc032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:30.347 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.355+0000 7f4e16153700 1 --2- 172.21.15.59:0/484498982 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f4dfc02fe60 0x7f4dfc032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f4e040096a0 tx=0x7f4e04006b40).stop 2024-01-13T11:49:30.347 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.355+0000 7f4e16153700 1 -- 172.21.15.59:0/484498982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e100fab80 msgr2=0x7f4e10108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:30.347 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.355+0000 7f4e16153700 1 --2- 172.21.15.59:0/484498982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e100fab80 0x7f4e10108770 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f4e00006a40 tx=0x7f4e00006f70).stop 2024-01-13T11:49:30.347 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.356+0000 7f4e16153700 1 -- 172.21.15.59:0/484498982 shutdown_connections 2024-01-13T11:49:30.347 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.356+0000 7f4e16153700 1 --2- 172.21.15.59:0/484498982 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f4dfc02fe60 0x7f4dfc032320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:30.347 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.356+0000 7f4e16153700 1 --2- 172.21.15.59:0/484498982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e100fab80 0x7f4e10108770 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:30.347 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.356+0000 7f4e16153700 1 -- 172.21.15.59:0/484498982 >> 172.21.15.59:0/484498982 conn(0x7f4e100f63c0 msgr2=0x7f4e100ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:30.347 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.356+0000 7f4e16153700 1 -- 172.21.15.59:0/484498982 shutdown_connections 2024-01-13T11:49:30.347 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:30.356+0000 7f4e16153700 1 -- 172.21.15.59:0/484498982 wait complete. 2024-01-13T11:49:30.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:30 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:29.386203+0000 mgr.smithi059.hmyldj (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:49:30.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:30 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cephadm 2024-01-13T11:49:29.386749+0000 mgr.smithi059.hmyldj (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-01-13T11:49:30.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:30 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:29.398984+0000 mon.smithi059 (mon.0) 79 : cluster [DBG] mgrmap e12: smithi059.hmyldj(active, since 2s) 2024-01-13T11:49:30.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:30 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:30.355461+0000 mon.smithi059 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.59:0/2761322198' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:30.708 INFO:teuthology.orchestra.run.smithi059.stderr:Deploying alertmanager service with default placement... 2024-01-13T11:49:30.708 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-01-13T11:49:31.113 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.122+0000 7fc49c8ce700 1 Processor -- start 2024-01-13T11:49:31.113 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.122+0000 7fc49c8ce700 1 -- start start 2024-01-13T11:49:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.122+0000 7fc49c8ce700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4980703d0 0x7fc4980fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.122+0000 7fc49c8ce700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4980fee30 con 0x7fc4980703d0 2024-01-13T11:49:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.123+0000 7fc49659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4980703d0 0x7fc4980fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.123+0000 7fc49659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4980703d0 0x7fc4980fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60112/0 (socket says 172.21.15.59:60112) 2024-01-13T11:49:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.123+0000 7fc49659c700 1 -- 172.21.15.59:0/3848015702 learned_addr learned my addr 172.21.15.59:0/3848015702 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.123+0000 7fc49659c700 1 -- 172.21.15.59:0/3848015702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4980fef70 con 0x7fc4980703d0 2024-01-13T11:49:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.123+0000 7fc49659c700 1 --2- 172.21.15.59:0/3848015702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4980703d0 0x7fc4980fe940 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fc48c00b5f0 tx=0x7fc48c006ee0).ready entity=mon.0 client_cookie=5f9a7dce310a71db server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.123+0000 7fc49559a700 1 -- 172.21.15.59:0/3848015702 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc48c00d600 con 0x7fc4980703d0 2024-01-13T11:49:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.123+0000 7fc49559a700 1 -- 172.21.15.59:0/3848015702 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc48c00dbd0 con 0x7fc4980703d0 2024-01-13T11:49:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.123+0000 7fc49559a700 1 -- 172.21.15.59:0/3848015702 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc48c011c00 con 0x7fc4980703d0 2024-01-13T11:49:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.124+0000 7fc49c8ce700 1 -- 172.21.15.59:0/3848015702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4980703d0 msgr2=0x7fc4980fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:31.116 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.124+0000 7fc49c8ce700 1 --2- 172.21.15.59:0/3848015702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4980703d0 0x7fc4980fe940 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fc48c00b5f0 tx=0x7fc48c006ee0).stop 2024-01-13T11:49:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.124+0000 7fc49c8ce700 1 -- 172.21.15.59:0/3848015702 shutdown_connections 2024-01-13T11:49:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.124+0000 7fc49c8ce700 1 --2- 172.21.15.59:0/3848015702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4980703d0 0x7fc4980fe940 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.124+0000 7fc49c8ce700 1 -- 172.21.15.59:0/3848015702 >> 172.21.15.59:0/3848015702 conn(0x7fc4980f63c0 msgr2=0x7fc4980f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.124+0000 7fc49c8ce700 1 -- 172.21.15.59:0/3848015702 shutdown_connections 2024-01-13T11:49:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.124+0000 7fc49c8ce700 1 -- 172.21.15.59:0/3848015702 wait complete. 2024-01-13T11:49:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.125+0000 7fc49c8ce700 1 Processor -- start 2024-01-13T11:49:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.125+0000 7fc49c8ce700 1 -- start start 2024-01-13T11:49:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.125+0000 7fc49c8ce700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4980703d0 0x7fc49810a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.125+0000 7fc49c8ce700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49810aec0 con 0x7fc4980703d0 2024-01-13T11:49:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.125+0000 7fc49659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4980703d0 0x7fc49810a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.125+0000 7fc49659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4980703d0 0x7fc49810a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60126/0 (socket says 172.21.15.59:60126) 2024-01-13T11:49:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.125+0000 7fc49659c700 1 -- 172.21.15.59:0/2530775210 learned_addr learned my addr 172.21.15.59:0/2530775210 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:31.117 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.125+0000 7fc49659c700 1 -- 172.21.15.59:0/2530775210 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48c00b200 con 0x7fc4980703d0 2024-01-13T11:49:31.118 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.125+0000 7fc49659c700 1 --2- 172.21.15.59:0/2530775210 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4980703d0 0x7fc49810a9d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fc48c00b190 tx=0x7fc48c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:31.118 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.126+0000 7fc486ffd700 1 -- 172.21.15.59:0/2530775210 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc48c01b410 con 0x7fc4980703d0 2024-01-13T11:49:31.119 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.126+0000 7fc486ffd700 1 -- 172.21.15.59:0/2530775210 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc48c01b9e0 con 0x7fc4980703d0 2024-01-13T11:49:31.119 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.126+0000 7fc49c8ce700 1 -- 172.21.15.59:0/2530775210 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc49810b0c0 con 0x7fc4980703d0 2024-01-13T11:49:31.119 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.126+0000 7fc486ffd700 1 -- 172.21.15.59:0/2530775210 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc48c01a880 con 0x7fc4980703d0 2024-01-13T11:49:31.119 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.126+0000 7fc49c8ce700 1 -- 172.21.15.59:0/2530775210 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc49810b4d0 con 0x7fc4980703d0 2024-01-13T11:49:31.119 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.126+0000 7fc486ffd700 1 -- 172.21.15.59:0/2530775210 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fc48c00f070 con 0x7fc4980703d0 2024-01-13T11:49:31.119 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.127+0000 7fc49c8ce700 1 -- 172.21.15.59:0/2530775210 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc498045ad0 con 0x7fc4980703d0 2024-01-13T11:49:31.119 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.127+0000 7fc486ffd700 1 --2- 172.21.15.59:0/2530775210 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fc47c02feb0 0x7fc47c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:31.119 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.127+0000 7fc486ffd700 1 -- 172.21.15.59:0/2530775210 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fc48c041330 con 0x7fc4980703d0 2024-01-13T11:49:31.119 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.127+0000 7fc495d9b700 1 --2- 172.21.15.59:0/2530775210 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fc47c02feb0 0x7fc47c032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:31.119 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.127+0000 7fc495d9b700 1 --2- 172.21.15.59:0/2530775210 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fc47c02feb0 0x7fc47c032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fc4880096a0 tx=0x7fc488006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:31.121 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.131+0000 7fc486ffd700 1 -- 172.21.15.59:0/2530775210 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc48c019950 con 0x7fc4980703d0 2024-01-13T11:49:31.306 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.315+0000 7fc49c8ce700 1 -- 172.21.15.59:0/2530775210 --> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fc49810be10 con 0x7fc47c02feb0 2024-01-13T11:49:31.310 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.319+0000 7fc486ffd700 1 -- 172.21.15.59:0/2530775210 <== mgr.14142 v2:172.21.15.59:6800/373398304 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fc49810be10 con 0x7fc47c02feb0 2024-01-13T11:49:31.310 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-01-13T11:49:31.313 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.321+0000 7fc49c8ce700 1 -- 172.21.15.59:0/2530775210 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fc47c02feb0 msgr2=0x7fc47c032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:31.313 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.321+0000 7fc49c8ce700 1 --2- 172.21.15.59:0/2530775210 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fc47c02feb0 0x7fc47c032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fc4880096a0 tx=0x7fc488006b40).stop 2024-01-13T11:49:31.313 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.321+0000 7fc49c8ce700 1 -- 172.21.15.59:0/2530775210 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4980703d0 msgr2=0x7fc49810a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:31.313 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.321+0000 7fc49c8ce700 1 --2- 172.21.15.59:0/2530775210 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4980703d0 0x7fc49810a9d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fc48c00b190 tx=0x7fc48c006ee0).stop 2024-01-13T11:49:31.314 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.321+0000 7fc49c8ce700 1 -- 172.21.15.59:0/2530775210 shutdown_connections 2024-01-13T11:49:31.314 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.321+0000 7fc49c8ce700 1 --2- 172.21.15.59:0/2530775210 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fc47c02feb0 0x7fc47c032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:31.314 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.321+0000 7fc49c8ce700 1 --2- 172.21.15.59:0/2530775210 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4980703d0 0x7fc49810a9d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:31.314 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.321+0000 7fc49c8ce700 1 -- 172.21.15.59:0/2530775210 >> 172.21.15.59:0/2530775210 conn(0x7fc4980f63c0 msgr2=0x7fc4980f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:31.314 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.322+0000 7fc49c8ce700 1 -- 172.21.15.59:0/2530775210 shutdown_connections 2024-01-13T11:49:31.314 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:31.322+0000 7fc49c8ce700 1 -- 172.21.15.59:0/2530775210 wait complete. 2024-01-13T11:49:31.683 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-01-13T11:49:32.052 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.060+0000 7f58a5970700 1 Processor -- start 2024-01-13T11:49:32.052 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.060+0000 7f58a5970700 1 -- start start 2024-01-13T11:49:32.052 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.061+0000 7f58a5970700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a00fab90 0x7f58a00fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:32.052 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.061+0000 7f58a5970700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a00fb4a0 con 0x7f58a00fab90 2024-01-13T11:49:32.052 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.061+0000 7f589effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a00fab90 0x7f58a00fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:32.053 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.061+0000 7f589effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a00fab90 0x7f58a00fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60132/0 (socket says 172.21.15.59:60132) 2024-01-13T11:49:32.053 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.061+0000 7f589effd700 1 -- 172.21.15.59:0/2561958702 learned_addr learned my addr 172.21.15.59:0/2561958702 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:32.053 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.061+0000 7f589effd700 1 -- 172.21.15.59:0/2561958702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a00fb5e0 con 0x7f58a00fab90 2024-01-13T11:49:32.053 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.061+0000 7f589effd700 1 --2- 172.21.15.59:0/2561958702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a00fab90 0x7f58a00fafb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f589400b870 tx=0x7f5894006f70).ready entity=mon.0 client_cookie=12c3f44832a5c9b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:32.053 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.061+0000 7f589dffb700 1 -- 172.21.15.59:0/2561958702 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f589400da20 con 0x7f58a00fab90 2024-01-13T11:49:32.053 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.061+0000 7f589dffb700 1 -- 172.21.15.59:0/2561958702 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5894010470 con 0x7f58a00fab90 2024-01-13T11:49:32.055 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.061+0000 7f589dffb700 1 -- 172.21.15.59:0/2561958702 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f589400f5b0 con 0x7f58a00fab90 2024-01-13T11:49:32.055 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.062+0000 7f58a5970700 1 -- 172.21.15.59:0/2561958702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a00fab90 msgr2=0x7f58a00fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:32.055 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.062+0000 7f58a5970700 1 --2- 172.21.15.59:0/2561958702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a00fab90 0x7f58a00fafb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f589400b870 tx=0x7f5894006f70).stop 2024-01-13T11:49:32.055 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.062+0000 7f58a5970700 1 -- 172.21.15.59:0/2561958702 shutdown_connections 2024-01-13T11:49:32.055 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.062+0000 7f58a5970700 1 --2- 172.21.15.59:0/2561958702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a00fab90 0x7f58a00fafb0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:32.055 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.062+0000 7f58a5970700 1 -- 172.21.15.59:0/2561958702 >> 172.21.15.59:0/2561958702 conn(0x7f58a00f63b0 msgr2=0x7f58a00f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:32.055 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.062+0000 7f58a5970700 1 -- 172.21.15.59:0/2561958702 shutdown_connections 2024-01-13T11:49:32.055 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.062+0000 7f58a5970700 1 -- 172.21.15.59:0/2561958702 wait complete. 2024-01-13T11:49:32.056 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.063+0000 7f58a5970700 1 Processor -- start 2024-01-13T11:49:32.056 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.063+0000 7f58a5970700 1 -- start start 2024-01-13T11:49:32.056 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.063+0000 7f58a5970700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a00fab90 0x7f58a0108770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:32.056 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.063+0000 7f58a5970700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a0108c60 con 0x7f58a00fab90 2024-01-13T11:49:32.056 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.063+0000 7f589effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a00fab90 0x7f58a0108770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:32.056 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.063+0000 7f589effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a00fab90 0x7f58a0108770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60148/0 (socket says 172.21.15.59:60148) 2024-01-13T11:49:32.056 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.063+0000 7f589effd700 1 -- 172.21.15.59:0/302793671 learned_addr learned my addr 172.21.15.59:0/302793671 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:32.056 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.064+0000 7f589effd700 1 -- 172.21.15.59:0/302793671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589400b480 con 0x7f58a00fab90 2024-01-13T11:49:32.056 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.064+0000 7f589effd700 1 --2- 172.21.15.59:0/302793671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a00fab90 0x7f58a0108770 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f5894006a40 tx=0x7f5894006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:32.056 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.064+0000 7f58877fe700 1 -- 172.21.15.59:0/302793671 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f589400f3d0 con 0x7f58a00fab90 2024-01-13T11:49:32.056 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.064+0000 7f58a5970700 1 -- 172.21.15.59:0/302793671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a0108e60 con 0x7f58a00fab90 2024-01-13T11:49:32.056 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.064+0000 7f58877fe700 1 -- 172.21.15.59:0/302793671 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f589400fef0 con 0x7f58a00fab90 2024-01-13T11:49:32.056 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.064+0000 7f58877fe700 1 -- 172.21.15.59:0/302793671 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5894019d40 con 0x7f58a00fab90 2024-01-13T11:49:32.057 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.064+0000 7f58a5970700 1 -- 172.21.15.59:0/302793671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a0109270 con 0x7f58a00fab90 2024-01-13T11:49:32.057 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.065+0000 7f58877fe700 1 -- 172.21.15.59:0/302793671 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f589402b060 con 0x7f58a00fab90 2024-01-13T11:49:32.057 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.065+0000 7f58877fe700 1 --2- 172.21.15.59:0/302793671 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f588c02feb0 0x7f588c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:32.057 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.065+0000 7f58a5970700 1 -- 172.21.15.59:0/302793671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a0045ad0 con 0x7f58a00fab90 2024-01-13T11:49:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.065+0000 7f58877fe700 1 -- 172.21.15.59:0/302793671 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f58940442c0 con 0x7f58a00fab90 2024-01-13T11:49:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.065+0000 7f589e7fc700 1 --2- 172.21.15.59:0/302793671 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f588c02feb0 0x7f588c032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.065+0000 7f589e7fc700 1 --2- 172.21.15.59:0/302793671 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f588c02feb0 0x7f588c032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f58900096a0 tx=0x7f5890006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:32.060 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.069+0000 7f58877fe700 1 -- 172.21.15.59:0/302793671 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f589400fa80 con 0x7f58a00fab90 2024-01-13T11:49:32.218 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.228+0000 7f58a5970700 1 -- 172.21.15.59:0/302793671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f58a00fc8a0 con 0x7f58a00fab90 2024-01-13T11:49:32.224 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.233+0000 7f58877fe700 1 -- 172.21.15.59:0/302793671 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f5894014030 con 0x7f58a00fab90 2024-01-13T11:49:32.226 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.235+0000 7f58a5970700 1 -- 172.21.15.59:0/302793671 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f588c02feb0 msgr2=0x7f588c032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:32.226 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.235+0000 7f58a5970700 1 --2- 172.21.15.59:0/302793671 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f588c02feb0 0x7f588c032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f58900096a0 tx=0x7f5890006b40).stop 2024-01-13T11:49:32.227 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.235+0000 7f58a5970700 1 -- 172.21.15.59:0/302793671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a00fab90 msgr2=0x7f58a0108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:32.227 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.235+0000 7f58a5970700 1 --2- 172.21.15.59:0/302793671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a00fab90 0x7f58a0108770 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f5894006a40 tx=0x7f5894006f70).stop 2024-01-13T11:49:32.227 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.235+0000 7f58a5970700 1 -- 172.21.15.59:0/302793671 shutdown_connections 2024-01-13T11:49:32.227 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.235+0000 7f58a5970700 1 --2- 172.21.15.59:0/302793671 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f588c02feb0 0x7f588c032370 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:32.227 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.235+0000 7f58a5970700 1 --2- 172.21.15.59:0/302793671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58a00fab90 0x7f58a0108770 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:32.227 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.235+0000 7f58a5970700 1 -- 172.21.15.59:0/302793671 >> 172.21.15.59:0/302793671 conn(0x7f58a00f63b0 msgr2=0x7f58a00ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:32.227 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.236+0000 7f58a5970700 1 -- 172.21.15.59:0/302793671 shutdown_connections 2024-01-13T11:49:32.227 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.236+0000 7f58a5970700 1 -- 172.21.15.59:0/302793671 wait complete. 2024-01-13T11:49:32.543 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:32 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:30.352509+0000 mgr.smithi059.hmyldj (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:49:32.543 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:32 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cephadm 2024-01-13T11:49:30.353176+0000 mgr.smithi059.hmyldj (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-01-13T11:49:32.543 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:32 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:31.320766+0000 mon.smithi059 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.59:0/2761322198' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:32.583 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-01-13T11:49:32.960 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.962+0000 7f0616d19700 1 Processor -- start 2024-01-13T11:49:32.960 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.962+0000 7f0616d19700 1 -- start start 2024-01-13T11:49:32.961 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.962+0000 7f0616d19700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06100703b0 0x7f0610100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:32.961 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.962+0000 7f0616d19700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06100fc5a0 con 0x7f06100703b0 2024-01-13T11:49:32.961 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.962+0000 7f0615d17700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06100703b0 0x7f0610100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:32.961 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.962+0000 7f0615d17700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06100703b0 0x7f0610100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60156/0 (socket says 172.21.15.59:60156) 2024-01-13T11:49:32.961 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.962+0000 7f0615d17700 1 -- 172.21.15.59:0/2600473868 learned_addr learned my addr 172.21.15.59:0/2600473868 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:32.961 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.963+0000 7f0615d17700 1 -- 172.21.15.59:0/2600473868 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06101010a0 con 0x7f06100703b0 2024-01-13T11:49:32.962 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.963+0000 7f0615d17700 1 --2- 172.21.15.59:0/2600473868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06100703b0 0x7f0610100bb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f060c00b5f0 tx=0x7f060c006ee0).ready entity=mon.0 client_cookie=17fafb255d112397 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:32.962 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.963+0000 7f0614d15700 1 -- 172.21.15.59:0/2600473868 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f060c00d600 con 0x7f06100703b0 2024-01-13T11:49:32.962 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.963+0000 7f0614d15700 1 -- 172.21.15.59:0/2600473868 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f060c00dbd0 con 0x7f06100703b0 2024-01-13T11:49:32.962 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.963+0000 7f0614d15700 1 -- 172.21.15.59:0/2600473868 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f060c011c00 con 0x7f06100703b0 2024-01-13T11:49:32.962 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.963+0000 7f0616d19700 1 -- 172.21.15.59:0/2600473868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06100703b0 msgr2=0x7f0610100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:32.962 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.963+0000 7f0616d19700 1 --2- 172.21.15.59:0/2600473868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06100703b0 0x7f0610100bb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f060c00b5f0 tx=0x7f060c006ee0).stop 2024-01-13T11:49:32.962 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.963+0000 7f0616d19700 1 -- 172.21.15.59:0/2600473868 shutdown_connections 2024-01-13T11:49:32.963 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.963+0000 7f0616d19700 1 --2- 172.21.15.59:0/2600473868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06100703b0 0x7f0610100bb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:32.963 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.963+0000 7f0616d19700 1 -- 172.21.15.59:0/2600473868 >> 172.21.15.59:0/2600473868 conn(0x7f06100f6210 msgr2=0x7f06100f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:32.963 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.964+0000 7f0616d19700 1 -- 172.21.15.59:0/2600473868 shutdown_connections 2024-01-13T11:49:32.963 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.964+0000 7f0616d19700 1 -- 172.21.15.59:0/2600473868 wait complete. 2024-01-13T11:49:32.963 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.964+0000 7f0616d19700 1 Processor -- start 2024-01-13T11:49:32.963 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.964+0000 7f0616d19700 1 -- start start 2024-01-13T11:49:32.963 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.965+0000 7f0616d19700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06100703b0 0x7f061010ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:32.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.965+0000 7f0616d19700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061010cf60 con 0x7f06100703b0 2024-01-13T11:49:32.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.965+0000 7f0615d17700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06100703b0 0x7f061010ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:32.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.965+0000 7f0615d17700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06100703b0 0x7f061010ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60158/0 (socket says 172.21.15.59:60158) 2024-01-13T11:49:32.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.965+0000 7f0615d17700 1 -- 172.21.15.59:0/2864423402 learned_addr learned my addr 172.21.15.59:0/2864423402 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:32.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.965+0000 7f0615d17700 1 -- 172.21.15.59:0/2864423402 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060c00b200 con 0x7f06100703b0 2024-01-13T11:49:32.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.965+0000 7f0615d17700 1 --2- 172.21.15.59:0/2864423402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06100703b0 0x7f061010ca70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f060c00e040 tx=0x7f060c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:32.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.965+0000 7f05fe7fc700 1 -- 172.21.15.59:0/2864423402 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f060c00d9f0 con 0x7f06100703b0 2024-01-13T11:49:32.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.965+0000 7f05fe7fc700 1 -- 172.21.15.59:0/2864423402 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f060c01a9f0 con 0x7f06100703b0 2024-01-13T11:49:32.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.965+0000 7f0616d19700 1 -- 172.21.15.59:0/2864423402 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061010d160 con 0x7f06100703b0 2024-01-13T11:49:32.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.965+0000 7f05fe7fc700 1 -- 172.21.15.59:0/2864423402 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f060c01b8c0 con 0x7f06100703b0 2024-01-13T11:49:32.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.965+0000 7f0616d19700 1 -- 172.21.15.59:0/2864423402 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f061010d570 con 0x7f06100703b0 2024-01-13T11:49:32.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.966+0000 7f05fe7fc700 1 -- 172.21.15.59:0/2864423402 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f060c00f070 con 0x7f06100703b0 2024-01-13T11:49:32.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.966+0000 7f05fe7fc700 1 --2- 172.21.15.59:0/2864423402 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f05f802feb0 0x7f05f8032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:32.966 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.966+0000 7f0616d19700 1 -- 172.21.15.59:0/2864423402 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0610045ad0 con 0x7f06100703b0 2024-01-13T11:49:32.966 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.966+0000 7f05fe7fc700 1 -- 172.21.15.59:0/2864423402 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f060c0414c0 con 0x7f06100703b0 2024-01-13T11:49:32.966 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.967+0000 7f0615516700 1 --2- 172.21.15.59:0/2864423402 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f05f802feb0 0x7f05f8032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:32.966 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.967+0000 7f0615516700 1 --2- 172.21.15.59:0/2864423402 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f05f802feb0 0x7f05f8032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f06000096a0 tx=0x7f0600006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:32.967 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:32.971+0000 7f05fe7fc700 1 -- 172.21.15.59:0/2864423402 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f060c006a70 con 0x7f06100703b0 2024-01-13T11:49:33.119 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.128+0000 7f0616d19700 1 -- 172.21.15.59:0/2864423402 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f0610101c00 con 0x7f06100703b0 2024-01-13T11:49:33.121 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.131+0000 7f05fe7fc700 1 -- 172.21.15.59:0/2864423402 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7f060c022310 con 0x7f06100703b0 2024-01-13T11:49:33.125 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.133+0000 7f0616d19700 1 -- 172.21.15.59:0/2864423402 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f05f802feb0 msgr2=0x7f05f8032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:33.125 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.133+0000 7f0616d19700 1 --2- 172.21.15.59:0/2864423402 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f05f802feb0 0x7f05f8032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f06000096a0 tx=0x7f0600006b40).stop 2024-01-13T11:49:33.125 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.133+0000 7f0616d19700 1 -- 172.21.15.59:0/2864423402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06100703b0 msgr2=0x7f061010ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:33.126 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.133+0000 7f0616d19700 1 --2- 172.21.15.59:0/2864423402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06100703b0 0x7f061010ca70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f060c00e040 tx=0x7f060c006ee0).stop 2024-01-13T11:49:33.126 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.133+0000 7f0616d19700 1 -- 172.21.15.59:0/2864423402 shutdown_connections 2024-01-13T11:49:33.126 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.133+0000 7f0616d19700 1 --2- 172.21.15.59:0/2864423402 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f05f802feb0 0x7f05f8032370 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:33.126 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.133+0000 7f0616d19700 1 --2- 172.21.15.59:0/2864423402 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06100703b0 0x7f061010ca70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:33.126 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.133+0000 7f0616d19700 1 -- 172.21.15.59:0/2864423402 >> 172.21.15.59:0/2864423402 conn(0x7f06100f6210 msgr2=0x7f06100f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:33.126 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.133+0000 7f0616d19700 1 -- 172.21.15.59:0/2864423402 shutdown_connections 2024-01-13T11:49:33.126 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.133+0000 7f0616d19700 1 -- 172.21.15.59:0/2864423402 wait complete. 2024-01-13T11:49:33.516 INFO:teuthology.orchestra.run.smithi059.stderr:Enabling the dashboard module... 2024-01-13T11:49:33.516 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-01-13T11:49:33.518 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:33 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:31.317930+0000 mgr.smithi059.hmyldj (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:49:33.518 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:33 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cephadm 2024-01-13T11:49:31.318477+0000 mgr.smithi059.hmyldj (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-01-13T11:49:33.518 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:33 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:32.235026+0000 mon.smithi059 (mon.0) 82 : audit [INF] from='client.? 172.21.15.59:0/302793671' entity='client.admin' 2024-01-13T11:49:33.518 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:33 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:33.132597+0000 mon.smithi059 (mon.0) 83 : audit [INF] from='client.? 172.21.15.59:0/2864423402' entity='client.admin' 2024-01-13T11:49:34.086 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.924+0000 7ff081ea7700 1 Processor -- start 2024-01-13T11:49:34.086 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.925+0000 7ff081ea7700 1 -- start start 2024-01-13T11:49:34.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.925+0000 7ff081ea7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff07c0703b0 0x7ff07c100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:34.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.925+0000 7ff081ea7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c0fc750 con 0x7ff07c0703b0 2024-01-13T11:49:34.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.925+0000 7ff07b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff07c0703b0 0x7ff07c100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:34.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.925+0000 7ff07b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff07c0703b0 0x7ff07c100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60170/0 (socket says 172.21.15.59:60170) 2024-01-13T11:49:34.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.925+0000 7ff07b7fe700 1 -- 172.21.15.59:0/987377773 learned_addr learned my addr 172.21.15.59:0/987377773 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:34.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.925+0000 7ff07b7fe700 1 -- 172.21.15.59:0/987377773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07c101250 con 0x7ff07c0703b0 2024-01-13T11:49:34.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.925+0000 7ff07b7fe700 1 --2- 172.21.15.59:0/987377773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff07c0703b0 0x7ff07c100d60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ff06c00b5f0 tx=0x7ff06c006ee0).ready entity=mon.0 client_cookie=ee52599107a6dd82 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:34.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.925+0000 7ff07a7fc700 1 -- 172.21.15.59:0/987377773 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff06c00d600 con 0x7ff07c0703b0 2024-01-13T11:49:34.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.925+0000 7ff07a7fc700 1 -- 172.21.15.59:0/987377773 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff06c00dbd0 con 0x7ff07c0703b0 2024-01-13T11:49:34.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.925+0000 7ff07a7fc700 1 -- 172.21.15.59:0/987377773 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff06c011c00 con 0x7ff07c0703b0 2024-01-13T11:49:34.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.926+0000 7ff081ea7700 1 -- 172.21.15.59:0/987377773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff07c0703b0 msgr2=0x7ff07c100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:34.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.926+0000 7ff081ea7700 1 --2- 172.21.15.59:0/987377773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff07c0703b0 0x7ff07c100d60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ff06c00b5f0 tx=0x7ff06c006ee0).stop 2024-01-13T11:49:34.088 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.926+0000 7ff081ea7700 1 -- 172.21.15.59:0/987377773 shutdown_connections 2024-01-13T11:49:34.088 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.926+0000 7ff081ea7700 1 --2- 172.21.15.59:0/987377773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff07c0703b0 0x7ff07c100d60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:34.088 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.926+0000 7ff081ea7700 1 -- 172.21.15.59:0/987377773 >> 172.21.15.59:0/987377773 conn(0x7ff07c0f63c0 msgr2=0x7ff07c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:34.088 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.926+0000 7ff081ea7700 1 -- 172.21.15.59:0/987377773 shutdown_connections 2024-01-13T11:49:34.088 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.926+0000 7ff081ea7700 1 -- 172.21.15.59:0/987377773 wait complete. 2024-01-13T11:49:34.088 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.926+0000 7ff081ea7700 1 Processor -- start 2024-01-13T11:49:34.088 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.927+0000 7ff081ea7700 1 -- start start 2024-01-13T11:49:34.088 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.927+0000 7ff081ea7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff07c0703b0 0x7ff07c10ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:34.088 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.927+0000 7ff081ea7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c10d1e0 con 0x7ff07c0703b0 2024-01-13T11:49:34.088 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.927+0000 7ff07b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff07c0703b0 0x7ff07c10ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:34.088 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.927+0000 7ff07b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff07c0703b0 0x7ff07c10ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60186/0 (socket says 172.21.15.59:60186) 2024-01-13T11:49:34.088 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.927+0000 7ff07b7fe700 1 -- 172.21.15.59:0/3259564014 learned_addr learned my addr 172.21.15.59:0/3259564014 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:34.088 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.927+0000 7ff07b7fe700 1 -- 172.21.15.59:0/3259564014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06c00b200 con 0x7ff07c0703b0 2024-01-13T11:49:34.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.927+0000 7ff07b7fe700 1 --2- 172.21.15.59:0/3259564014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff07c0703b0 0x7ff07c10ccf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7ff06c00e040 tx=0x7ff06c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:34.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.927+0000 7ff073fff700 1 -- 172.21.15.59:0/3259564014 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff06c00d9f0 con 0x7ff07c0703b0 2024-01-13T11:49:34.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.927+0000 7ff073fff700 1 -- 172.21.15.59:0/3259564014 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff06c01a9f0 con 0x7ff07c0703b0 2024-01-13T11:49:34.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.928+0000 7ff081ea7700 1 -- 172.21.15.59:0/3259564014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07c10d3e0 con 0x7ff07c0703b0 2024-01-13T11:49:34.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.928+0000 7ff073fff700 1 -- 172.21.15.59:0/3259564014 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff06c01b8c0 con 0x7ff07c0703b0 2024-01-13T11:49:34.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.928+0000 7ff081ea7700 1 -- 172.21.15.59:0/3259564014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07c10d7f0 con 0x7ff07c0703b0 2024-01-13T11:49:34.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.928+0000 7ff073fff700 1 -- 172.21.15.59:0/3259564014 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7ff06c00f070 con 0x7ff07c0703b0 2024-01-13T11:49:34.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.928+0000 7ff073fff700 1 --2- 172.21.15.59:0/3259564014 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7ff05c02fe60 0x7ff05c032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:34.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.929+0000 7ff073fff700 1 -- 172.21.15.59:0/3259564014 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7ff06c041510 con 0x7ff07c0703b0 2024-01-13T11:49:34.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.929+0000 7ff081ea7700 1 -- 172.21.15.59:0/3259564014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff07c045ad0 con 0x7ff07c0703b0 2024-01-13T11:49:34.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.929+0000 7ff07affd700 1 --2- 172.21.15.59:0/3259564014 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7ff05c02fe60 0x7ff05c032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:34.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.929+0000 7ff07affd700 1 --2- 172.21.15.59:0/3259564014 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7ff05c02fe60 0x7ff05c032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff0640096a0 tx=0x7ff064006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:34.089 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:33.933+0000 7ff073fff700 1 -- 172.21.15.59:0/3259564014 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff06c016070 con 0x7ff07c0703b0 2024-01-13T11:49:34.120 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:34.130+0000 7ff081ea7700 1 -- 172.21.15.59:0/3259564014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7ff07c101db0 con 0x7ff07c0703b0 2024-01-13T11:49:34.177 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:34.186+0000 7ff073fff700 1 -- 172.21.15.59:0/3259564014 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7ff06c0257e0 con 0x7ff07c0703b0 2024-01-13T11:49:34.177 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:34.187+0000 7ff073fff700 1 -- 172.21.15.59:0/3259564014 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7ff06c010880 con 0x7ff07c0703b0 2024-01-13T11:49:34.180 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:34.189+0000 7ff081ea7700 1 -- 172.21.15.59:0/3259564014 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7ff05c02fe60 msgr2=0x7ff05c032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:34.180 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:34.189+0000 7ff081ea7700 1 --2- 172.21.15.59:0/3259564014 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7ff05c02fe60 0x7ff05c032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff0640096a0 tx=0x7ff064006b40).stop 2024-01-13T11:49:34.181 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:34.189+0000 7ff081ea7700 1 -- 172.21.15.59:0/3259564014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff07c0703b0 msgr2=0x7ff07c10ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:34.181 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:34.189+0000 7ff081ea7700 1 --2- 172.21.15.59:0/3259564014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff07c0703b0 0x7ff07c10ccf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7ff06c00e040 tx=0x7ff06c006ee0).stop 2024-01-13T11:49:34.181 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:34.189+0000 7ff081ea7700 1 -- 172.21.15.59:0/3259564014 shutdown_connections 2024-01-13T11:49:34.181 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:34.189+0000 7ff081ea7700 1 --2- 172.21.15.59:0/3259564014 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7ff05c02fe60 0x7ff05c032320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:34.181 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:34.189+0000 7ff081ea7700 1 --2- 172.21.15.59:0/3259564014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff07c0703b0 0x7ff07c10ccf0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:34.181 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:34.189+0000 7ff081ea7700 1 -- 172.21.15.59:0/3259564014 >> 172.21.15.59:0/3259564014 conn(0x7ff07c0f63c0 msgr2=0x7ff07c0f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:34.181 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:34.189+0000 7ff081ea7700 1 -- 172.21.15.59:0/3259564014 shutdown_connections 2024-01-13T11:49:34.181 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:34.189+0000 7ff081ea7700 1 -- 172.21.15.59:0/3259564014 wait complete. 2024-01-13T11:49:34.495 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:34 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:34.131674+0000 mon.smithi059 (mon.0) 84 : audit [INF] from='client.? 172.21.15.59:0/3259564014' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-13T11:49:34.549 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-13T11:49:34.994 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.001+0000 7fafcdd20700 1 Processor -- start 2024-01-13T11:49:34.994 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.001+0000 7fafcdd20700 1 -- start start 2024-01-13T11:49:34.994 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.001+0000 7fafcdd20700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc80703d0 0x7fafc80fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:34.994 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.001+0000 7fafcdd20700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc80fee30 con 0x7fafc80703d0 2024-01-13T11:49:34.995 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.002+0000 7fafc77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc80703d0 0x7fafc80fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:34.995 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.002+0000 7fafc77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc80703d0 0x7fafc80fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60220/0 (socket says 172.21.15.59:60220) 2024-01-13T11:49:34.995 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.002+0000 7fafc77fe700 1 -- 172.21.15.59:0/571111984 learned_addr learned my addr 172.21.15.59:0/571111984 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:34.995 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.002+0000 7fafc77fe700 1 -- 172.21.15.59:0/571111984 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc80fef70 con 0x7fafc80703d0 2024-01-13T11:49:34.995 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.002+0000 7fafc77fe700 1 --2- 172.21.15.59:0/571111984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc80703d0 0x7fafc80fe940 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fafb800b5f0 tx=0x7fafb8006ee0).ready entity=mon.0 client_cookie=7681b9c4a391021a server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:34.995 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.002+0000 7fafc67fc700 1 -- 172.21.15.59:0/571111984 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafb800d600 con 0x7fafc80703d0 2024-01-13T11:49:34.995 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.002+0000 7fafc67fc700 1 -- 172.21.15.59:0/571111984 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fafb800dbd0 con 0x7fafc80703d0 2024-01-13T11:49:34.995 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.002+0000 7fafc67fc700 1 -- 172.21.15.59:0/571111984 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafb8011c00 con 0x7fafc80703d0 2024-01-13T11:49:34.995 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.003+0000 7fafcdd20700 1 -- 172.21.15.59:0/571111984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc80703d0 msgr2=0x7fafc80fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:34.995 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.003+0000 7fafcdd20700 1 --2- 172.21.15.59:0/571111984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc80703d0 0x7fafc80fe940 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fafb800b5f0 tx=0x7fafb8006ee0).stop 2024-01-13T11:49:34.995 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.003+0000 7fafcdd20700 1 -- 172.21.15.59:0/571111984 shutdown_connections 2024-01-13T11:49:34.995 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.003+0000 7fafcdd20700 1 --2- 172.21.15.59:0/571111984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc80703d0 0x7fafc80fe940 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:34.995 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.003+0000 7fafcdd20700 1 -- 172.21.15.59:0/571111984 >> 172.21.15.59:0/571111984 conn(0x7fafc80f63c0 msgr2=0x7fafc80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:34.996 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.004+0000 7fafcdd20700 1 -- 172.21.15.59:0/571111984 shutdown_connections 2024-01-13T11:49:34.996 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.004+0000 7fafcdd20700 1 -- 172.21.15.59:0/571111984 wait complete. 2024-01-13T11:49:34.999 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.004+0000 7fafcdd20700 1 Processor -- start 2024-01-13T11:49:34.999 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.004+0000 7fafcdd20700 1 -- start start 2024-01-13T11:49:34.999 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.005+0000 7fafcdd20700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc80703d0 0x7fafc810a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:34.999 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.005+0000 7fafcdd20700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc810aec0 con 0x7fafc80703d0 2024-01-13T11:49:35.000 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.005+0000 7fafc77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc80703d0 0x7fafc810a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:35.000 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.005+0000 7fafc77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc80703d0 0x7fafc810a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60224/0 (socket says 172.21.15.59:60224) 2024-01-13T11:49:35.000 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.005+0000 7fafc77fe700 1 -- 172.21.15.59:0/3987652669 learned_addr learned my addr 172.21.15.59:0/3987652669 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:35.000 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.005+0000 7fafc77fe700 1 -- 172.21.15.59:0/3987652669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb800b200 con 0x7fafc80703d0 2024-01-13T11:49:35.000 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.005+0000 7fafc77fe700 1 --2- 172.21.15.59:0/3987652669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc80703d0 0x7fafc810a9d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fafb800b190 tx=0x7fafb8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:35.000 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.005+0000 7fafaffff700 1 -- 172.21.15.59:0/3987652669 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafb801b410 con 0x7fafc80703d0 2024-01-13T11:49:35.000 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.005+0000 7fafaffff700 1 -- 172.21.15.59:0/3987652669 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fafb801b9e0 con 0x7fafc80703d0 2024-01-13T11:49:35.000 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.005+0000 7fafcdd20700 1 -- 172.21.15.59:0/3987652669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc810b0c0 con 0x7fafc80703d0 2024-01-13T11:49:35.000 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.005+0000 7fafaffff700 1 -- 172.21.15.59:0/3987652669 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafb801a880 con 0x7fafc80703d0 2024-01-13T11:49:35.001 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.006+0000 7fafcdd20700 1 -- 172.21.15.59:0/3987652669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc810b4d0 con 0x7fafc80703d0 2024-01-13T11:49:35.001 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.006+0000 7fafaffff700 1 -- 172.21.15.59:0/3987652669 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fafb800f070 con 0x7fafc80703d0 2024-01-13T11:49:35.001 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.006+0000 7fafaffff700 1 --2- 172.21.15.59:0/3987652669 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fafb402ff00 0x7fafb40323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:35.001 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.007+0000 7fafaffff700 1 -- 172.21.15.59:0/3987652669 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fafb8042310 con 0x7fafc80703d0 2024-01-13T11:49:35.001 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.007+0000 7fafc6ffd700 1 -- 172.21.15.59:0/3987652669 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fafb402ff00 msgr2=0x7fafb40323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/373398304 2024-01-13T11:49:35.001 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.007+0000 7fafc6ffd700 1 --2- 172.21.15.59:0/3987652669 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fafb402ff00 0x7fafb40323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-13T11:49:35.001 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.008+0000 7fafcdd20700 1 -- 172.21.15.59:0/3987652669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc8059100 con 0x7fafc80703d0 2024-01-13T11:49:35.002 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.012+0000 7fafaffff700 1 -- 172.21.15.59:0/3987652669 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fafb8016070 con 0x7fafc80703d0 2024-01-13T11:49:35.198 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.208+0000 7fafc6ffd700 1 -- 172.21.15.59:0/3987652669 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fafb402ff00 msgr2=0x7fafb40323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/373398304 2024-01-13T11:49:35.199 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.208+0000 7fafc6ffd700 1 --2- 172.21.15.59:0/3987652669 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fafb402ff00 0x7fafb40323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-13T11:49:35.205 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.215+0000 7fafcdd20700 1 -- 172.21.15.59:0/3987652669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fafc81047f0 con 0x7fafc80703d0 2024-01-13T11:49:35.206 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.215+0000 7fafaffff700 1 -- 172.21.15.59:0/3987652669 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7fafb8025460 con 0x7fafc80703d0 2024-01-13T11:49:35.206 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: { 2024-01-13T11:49:35.206 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 13, 2024-01-13T11:49:35.206 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "available": true, 2024-01-13T11:49:35.206 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "active_name": "smithi059.hmyldj", 2024-01-13T11:49:35.207 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-13T11:49:35.207 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: } 2024-01-13T11:49:35.208 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.217+0000 7fafcdd20700 1 -- 172.21.15.59:0/3987652669 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fafb402ff00 msgr2=0x7fafb40323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-13T11:49:35.208 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.217+0000 7fafcdd20700 1 --2- 172.21.15.59:0/3987652669 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fafb402ff00 0x7fafb40323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-13T11:49:35.209 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.217+0000 7fafcdd20700 1 -- 172.21.15.59:0/3987652669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc80703d0 msgr2=0x7fafc810a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:35.209 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.217+0000 7fafcdd20700 1 --2- 172.21.15.59:0/3987652669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc80703d0 0x7fafc810a9d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fafb800b190 tx=0x7fafb8006ee0).stop 2024-01-13T11:49:35.209 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.217+0000 7fafcdd20700 1 -- 172.21.15.59:0/3987652669 shutdown_connections 2024-01-13T11:49:35.209 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.217+0000 7fafcdd20700 1 --2- 172.21.15.59:0/3987652669 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7fafb402ff00 0x7fafb40323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-13T11:49:35.209 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.217+0000 7fafcdd20700 1 --2- 172.21.15.59:0/3987652669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafc80703d0 0x7fafc810a9d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:35.209 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.217+0000 7fafcdd20700 1 -- 172.21.15.59:0/3987652669 >> 172.21.15.59:0/3987652669 conn(0x7fafc80f63c0 msgr2=0x7fafc80f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:35.209 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.217+0000 7fafcdd20700 1 -- 172.21.15.59:0/3987652669 shutdown_connections 2024-01-13T11:49:35.209 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:35.217+0000 7fafcdd20700 1 -- 172.21.15.59:0/3987652669 wait complete. 2024-01-13T11:49:35.380 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:35 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:34.188977+0000 mon.smithi059 (mon.0) 85 : audit [INF] from='client.? 172.21.15.59:0/3259564014' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-13T11:49:35.380 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:35 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:34.189044+0000 mon.smithi059 (mon.0) 86 : cluster [DBG] mgrmap e13: smithi059.hmyldj(active, since 6s) 2024-01-13T11:49:35.659 INFO:teuthology.orchestra.run.smithi059.stderr:Waiting for the mgr to restart... 2024-01-13T11:49:35.659 INFO:teuthology.orchestra.run.smithi059.stderr:Waiting for mgr epoch 13... 2024-01-13T11:49:35.660 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-13T11:49:36.147 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.154+0000 7f7770072700 1 Processor -- start 2024-01-13T11:49:36.147 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.154+0000 7f7770072700 1 -- start start 2024-01-13T11:49:36.147 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.154+0000 7f7770072700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77680703b0 0x7f7768100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:36.147 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.154+0000 7f7770072700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77680fc750 con 0x7f77680703b0 2024-01-13T11:49:36.147 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.154+0000 7f776de0e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77680703b0 0x7f7768100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:36.147 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.154+0000 7f776de0e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77680703b0 0x7f7768100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60228/0 (socket says 172.21.15.59:60228) 2024-01-13T11:49:36.147 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.154+0000 7f776de0e700 1 -- 172.21.15.59:0/398132765 learned_addr learned my addr 172.21.15.59:0/398132765 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:36.147 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.155+0000 7f776de0e700 1 -- 172.21.15.59:0/398132765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7768101250 con 0x7f77680703b0 2024-01-13T11:49:36.147 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.155+0000 7f776de0e700 1 --2- 172.21.15.59:0/398132765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77680703b0 0x7f7768100d60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f776400b5f0 tx=0x7f7764006ee0).ready entity=mon.0 client_cookie=e4298032309c60a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:36.147 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.155+0000 7f776ce0c700 1 -- 172.21.15.59:0/398132765 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f776400d600 con 0x7f77680703b0 2024-01-13T11:49:36.148 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.155+0000 7f776ce0c700 1 -- 172.21.15.59:0/398132765 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f776400dbd0 con 0x7f77680703b0 2024-01-13T11:49:36.148 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.155+0000 7f776ce0c700 1 -- 172.21.15.59:0/398132765 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7764011c00 con 0x7f77680703b0 2024-01-13T11:49:36.148 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.155+0000 7f7770072700 1 -- 172.21.15.59:0/398132765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77680703b0 msgr2=0x7f7768100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:36.148 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.155+0000 7f7770072700 1 --2- 172.21.15.59:0/398132765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77680703b0 0x7f7768100d60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f776400b5f0 tx=0x7f7764006ee0).stop 2024-01-13T11:49:36.148 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.156+0000 7f7770072700 1 -- 172.21.15.59:0/398132765 shutdown_connections 2024-01-13T11:49:36.148 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.156+0000 7f7770072700 1 --2- 172.21.15.59:0/398132765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77680703b0 0x7f7768100d60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:36.148 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.156+0000 7f7770072700 1 -- 172.21.15.59:0/398132765 >> 172.21.15.59:0/398132765 conn(0x7f77680f63c0 msgr2=0x7f77680f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:36.148 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.156+0000 7f7770072700 1 -- 172.21.15.59:0/398132765 shutdown_connections 2024-01-13T11:49:36.148 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.156+0000 7f7770072700 1 -- 172.21.15.59:0/398132765 wait complete. 2024-01-13T11:49:36.150 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.156+0000 7f7770072700 1 Processor -- start 2024-01-13T11:49:36.150 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.157+0000 7f7770072700 1 -- start start 2024-01-13T11:49:36.150 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.157+0000 7f7770072700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77680703b0 0x7f776810cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:36.151 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.157+0000 7f7770072700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776810d100 con 0x7f77680703b0 2024-01-13T11:49:36.151 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.157+0000 7f776de0e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77680703b0 0x7f776810cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:36.151 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.157+0000 7f776de0e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77680703b0 0x7f776810cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60232/0 (socket says 172.21.15.59:60232) 2024-01-13T11:49:36.151 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.157+0000 7f776de0e700 1 -- 172.21.15.59:0/3571294960 learned_addr learned my addr 172.21.15.59:0/3571294960 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:36.151 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.157+0000 7f776de0e700 1 -- 172.21.15.59:0/3571294960 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776400b200 con 0x7f77680703b0 2024-01-13T11:49:36.151 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.157+0000 7f776de0e700 1 --2- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77680703b0 0x7f776810cc10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f776400b190 tx=0x7f7764006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:36.151 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.157+0000 7f77567fc700 1 -- 172.21.15.59:0/3571294960 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f776401b410 con 0x7f77680703b0 2024-01-13T11:49:36.151 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.158+0000 7f77567fc700 1 -- 172.21.15.59:0/3571294960 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f776401b9e0 con 0x7f77680703b0 2024-01-13T11:49:36.151 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.158+0000 7f7770072700 1 -- 172.21.15.59:0/3571294960 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f776810d300 con 0x7f77680703b0 2024-01-13T11:49:36.151 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.158+0000 7f77567fc700 1 -- 172.21.15.59:0/3571294960 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f776401a880 con 0x7f77680703b0 2024-01-13T11:49:36.152 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.158+0000 7f7770072700 1 -- 172.21.15.59:0/3571294960 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f776810d710 con 0x7f77680703b0 2024-01-13T11:49:36.152 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.158+0000 7f77567fc700 1 -- 172.21.15.59:0/3571294960 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f776400f070 con 0x7f77680703b0 2024-01-13T11:49:36.152 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.159+0000 7f77567fc700 1 --2- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f775002feb0 0x7f7750032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:36.152 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.159+0000 7f77567fc700 1 -- 172.21.15.59:0/3571294960 --> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7750032a30 con 0x7f775002feb0 2024-01-13T11:49:36.152 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.159+0000 7f77567fc700 1 -- 172.21.15.59:0/3571294960 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f7764041530 con 0x7f77680703b0 2024-01-13T11:49:36.152 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.159+0000 7f776d60d700 1 -- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f775002feb0 msgr2=0x7f7750032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/373398304 2024-01-13T11:49:36.153 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.159+0000 7f776d60d700 1 --2- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f775002feb0 0x7f7750032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-13T11:49:36.350 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.360+0000 7f776d60d700 1 -- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f775002feb0 msgr2=0x7f7750032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/373398304 2024-01-13T11:49:36.351 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.360+0000 7f776d60d700 1 --2- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f775002feb0 0x7f7750032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-13T11:49:36.388 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:36 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:35.217110+0000 mon.smithi059 (mon.0) 87 : audit [DBG] from='client.? 172.21.15.59:0/3987652669' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-13T11:49:36.751 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.761+0000 7f776d60d700 1 -- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f775002feb0 msgr2=0x7f7750032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/373398304 2024-01-13T11:49:36.752 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:36.761+0000 7f776d60d700 1 --2- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f775002feb0 0x7f7750032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-13T11:49:37.552 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:37.562+0000 7f776d60d700 1 -- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f775002feb0 msgr2=0x7f7750032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/373398304 2024-01-13T11:49:37.552 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:37.562+0000 7f776d60d700 1 --2- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f775002feb0 0x7f7750032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-13T11:49:39.154 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:39.163+0000 7f776d60d700 1 -- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f775002feb0 msgr2=0x7f7750032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/373398304 2024-01-13T11:49:39.154 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:39.163+0000 7f776d60d700 1 --2- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f775002feb0 0x7f7750032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-13T11:49:39.486 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:39.495+0000 7f77567fc700 1 -- 172.21.15.59:0/3571294960 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f776401aaf0 con 0x7f77680703b0 2024-01-13T11:49:39.486 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:39.495+0000 7f77567fc700 1 -- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f775002feb0 msgr2=0x7f7750032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-13T11:49:39.486 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:39.495+0000 7f77567fc700 1 --2- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f775002feb0 0x7f7750032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-13T11:49:39.889 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:39 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:39.442322+0000 mon.smithi059 (mon.0) 88 : cluster [INF] Active manager daemon smithi059.hmyldj restarted 2024-01-13T11:49:39.889 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:39 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:39.442785+0000 mon.smithi059 (mon.0) 89 : cluster [INF] Activating manager daemon smithi059.hmyldj 2024-01-13T11:49:39.889 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:39 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:39.446629+0000 mon.smithi059 (mon.0) 90 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-01-13T11:49:40.494 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:40.502+0000 7f77567fc700 1 -- 172.21.15.59:0/3571294960 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f7764025460 con 0x7f77680703b0 2024-01-13T11:49:40.494 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:40.502+0000 7f77567fc700 1 --2- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7750033b40 0x7f7750035f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:40.494 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:40.502+0000 7f77567fc700 1 -- 172.21.15.59:0/3571294960 --> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7750032a30 con 0x7f7750033b40 2024-01-13T11:49:40.494 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:40.503+0000 7f776d60d700 1 --2- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7750033b40 0x7f7750035f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:40.495 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:40.503+0000 7f776d60d700 1 --2- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7750033b40 0x7f7750035f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f7758006fd0 tx=0x7f7758006cf0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:40.495 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:40.504+0000 7f77567fc700 1 -- 172.21.15.59:0/3571294960 <== mgr.14162 v2:172.21.15.59:6800/32893793 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f7750032a30 con 0x7f7750033b40 2024-01-13T11:49:40.502 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:40.511+0000 7f7770072700 1 -- 172.21.15.59:0/3571294960 --> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f7768059100 con 0x7f7750033b40 2024-01-13T11:49:40.502 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:40.511+0000 7f77567fc700 1 -- 172.21.15.59:0/3571294960 <== mgr.14162 v2:172.21.15.59:6800/32893793 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f7768059100 con 0x7f7750033b40 2024-01-13T11:49:40.502 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: { 2024-01-13T11:49:40.502 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-01-13T11:49:40.503 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "initialized": true 2024-01-13T11:49:40.503 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: } 2024-01-13T11:49:40.503 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:40.512+0000 7f7770072700 1 -- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7750033b40 msgr2=0x7f7750035f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:40.504 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:40.512+0000 7f7770072700 1 --2- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7750033b40 0x7f7750035f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f7758006fd0 tx=0x7f7758006cf0).stop 2024-01-13T11:49:40.504 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:40.512+0000 7f7770072700 1 -- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77680703b0 msgr2=0x7f776810cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:40.504 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:40.512+0000 7f7770072700 1 --2- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77680703b0 0x7f776810cc10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f776400b190 tx=0x7f7764006ee0).stop 2024-01-13T11:49:40.504 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:40.512+0000 7f7770072700 1 -- 172.21.15.59:0/3571294960 shutdown_connections 2024-01-13T11:49:40.504 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:40.512+0000 7f7770072700 1 --2- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7750033b40 0x7f7750035f30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:40.504 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:40.512+0000 7f7770072700 1 --2- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77680703b0 0x7f776810cc10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:40.504 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:40.512+0000 7f7770072700 1 --2- 172.21.15.59:0/3571294960 >> [v2:172.21.15.59:6800/373398304,v1:172.21.15.59:6801/373398304] conn(0x7f775002feb0 0x7f7750032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-13T11:49:40.505 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:40.512+0000 7f7770072700 1 -- 172.21.15.59:0/3571294960 >> 172.21.15.59:0/3571294960 conn(0x7f77680f63c0 msgr2=0x7f77680f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:40.505 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:40.512+0000 7f7770072700 1 -- 172.21.15.59:0/3571294960 shutdown_connections 2024-01-13T11:49:40.505 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:40.512+0000 7f7770072700 1 -- 172.21.15.59:0/3571294960 wait complete. 2024-01-13T11:49:40.835 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:39.497856+0000 mon.smithi059 (mon.0) 91 : cluster [DBG] mgrmap e14: smithi059.hmyldj(active, starting, since 0.0551609s) 2024-01-13T11:49:40.835 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:39.499660+0000 mon.smithi059 (mon.0) 92 : audit [DBG] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-01-13T11:49:40.835 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:39.500257+0000 mon.smithi059 (mon.0) 93 : audit [DBG] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "mgr metadata", "who": "smithi059.hmyldj", "id": "smithi059.hmyldj"}]: dispatch 2024-01-13T11:49:40.835 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:39.501010+0000 mon.smithi059 (mon.0) 94 : audit [DBG] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T11:49:40.835 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:39.501188+0000 mon.smithi059 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T11:49:40.835 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:39.503255+0000 mon.smithi059 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T11:49:40.835 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:39.505227+0000 mon.smithi059 (mon.0) 97 : cluster [INF] Manager daemon smithi059.hmyldj is now available 2024-01-13T11:49:40.835 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:39.609382+0000 mon.smithi059 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:49:40.836 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:39.610902+0000 mon.smithi059 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:49:40.836 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:39.635146+0000 mon.smithi059 (mon.0) 100 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/mirror_snapshot_schedule"}]: dispatch 2024-01-13T11:49:40.836 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:39.637756+0000 mon.smithi059 (mon.0) 101 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/trash_purge_schedule"}]: dispatch 2024-01-13T11:49:40.866 INFO:teuthology.orchestra.run.smithi059.stderr:mgr epoch 13 is available 2024-01-13T11:49:40.866 INFO:teuthology.orchestra.run.smithi059.stderr:Generating a dashboard self-signed certificate... 2024-01-13T11:49:40.866 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-01-13T11:49:41.208 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.217+0000 7f1f4b016700 1 Processor -- start 2024-01-13T11:49:41.211 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.217+0000 7f1f4b016700 1 -- start start 2024-01-13T11:49:41.211 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.217+0000 7f1f4b016700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f440704a0 0x7f1f440fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:41.211 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.217+0000 7f1f4b016700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f440fede0 con 0x7f1f440704a0 2024-01-13T11:49:41.211 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.217+0000 7f1f48db2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f440704a0 0x7f1f440fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:41.211 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.217+0000 7f1f48db2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f440704a0 0x7f1f440fe8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34824/0 (socket says 172.21.15.59:34824) 2024-01-13T11:49:41.211 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.217+0000 7f1f48db2700 1 -- 172.21.15.59:0/3224155752 learned_addr learned my addr 172.21.15.59:0/3224155752 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:41.211 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.218+0000 7f1f48db2700 1 -- 172.21.15.59:0/3224155752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f440fef20 con 0x7f1f440704a0 2024-01-13T11:49:41.211 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.218+0000 7f1f48db2700 1 --2- 172.21.15.59:0/3224155752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f440704a0 0x7f1f440fe8f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f1f3800b5f0 tx=0x7f1f38006ee0).ready entity=mon.0 client_cookie=621d1c64c1f34f6f server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:41.211 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.218+0000 7f1f437fe700 1 -- 172.21.15.59:0/3224155752 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f3800d600 con 0x7f1f440704a0 2024-01-13T11:49:41.211 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.218+0000 7f1f437fe700 1 -- 172.21.15.59:0/3224155752 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1f3800dbd0 con 0x7f1f440704a0 2024-01-13T11:49:41.211 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.218+0000 7f1f437fe700 1 -- 172.21.15.59:0/3224155752 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f38011c00 con 0x7f1f440704a0 2024-01-13T11:49:41.212 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.219+0000 7f1f4b016700 1 -- 172.21.15.59:0/3224155752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f440704a0 msgr2=0x7f1f440fe8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:41.212 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.219+0000 7f1f4b016700 1 --2- 172.21.15.59:0/3224155752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f440704a0 0x7f1f440fe8f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f1f3800b5f0 tx=0x7f1f38006ee0).stop 2024-01-13T11:49:41.212 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.219+0000 7f1f4b016700 1 -- 172.21.15.59:0/3224155752 shutdown_connections 2024-01-13T11:49:41.212 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.219+0000 7f1f4b016700 1 --2- 172.21.15.59:0/3224155752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f440704a0 0x7f1f440fe8f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:41.212 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.219+0000 7f1f4b016700 1 -- 172.21.15.59:0/3224155752 >> 172.21.15.59:0/3224155752 conn(0x7f1f440f63b0 msgr2=0x7f1f440f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:41.212 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.219+0000 7f1f4b016700 1 -- 172.21.15.59:0/3224155752 shutdown_connections 2024-01-13T11:49:41.212 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.219+0000 7f1f4b016700 1 -- 172.21.15.59:0/3224155752 wait complete. 2024-01-13T11:49:41.212 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.219+0000 7f1f4b016700 1 Processor -- start 2024-01-13T11:49:41.212 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.220+0000 7f1f4b016700 1 -- start start 2024-01-13T11:49:41.212 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.220+0000 7f1f4b016700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f440704a0 0x7f1f440fc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:41.212 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.220+0000 7f1f4b016700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f440fa7f0 con 0x7f1f440704a0 2024-01-13T11:49:41.212 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.220+0000 7f1f48db2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f440704a0 0x7f1f440fc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:41.212 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.220+0000 7f1f48db2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f440704a0 0x7f1f440fc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34832/0 (socket says 172.21.15.59:34832) 2024-01-13T11:49:41.213 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.220+0000 7f1f48db2700 1 -- 172.21.15.59:0/3762578329 learned_addr learned my addr 172.21.15.59:0/3762578329 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:41.214 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.220+0000 7f1f48db2700 1 -- 172.21.15.59:0/3762578329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3800b200 con 0x7f1f440704a0 2024-01-13T11:49:41.214 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.220+0000 7f1f48db2700 1 --2- 172.21.15.59:0/3762578329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f440704a0 0x7f1f440fc370 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f1f3800b190 tx=0x7f1f38006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:41.214 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.221+0000 7f1f417fa700 1 -- 172.21.15.59:0/3762578329 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f3801b410 con 0x7f1f440704a0 2024-01-13T11:49:41.214 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.221+0000 7f1f417fa700 1 -- 172.21.15.59:0/3762578329 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1f3801b9e0 con 0x7f1f440704a0 2024-01-13T11:49:41.214 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.221+0000 7f1f4b016700 1 -- 172.21.15.59:0/3762578329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f440fa990 con 0x7f1f440704a0 2024-01-13T11:49:41.214 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.221+0000 7f1f417fa700 1 -- 172.21.15.59:0/3762578329 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f3801a880 con 0x7f1f440704a0 2024-01-13T11:49:41.214 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.221+0000 7f1f4b016700 1 -- 172.21.15.59:0/3762578329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f440fada0 con 0x7f1f440704a0 2024-01-13T11:49:41.214 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.222+0000 7f1f417fa700 1 -- 172.21.15.59:0/3762578329 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f1f3800f070 con 0x7f1f440704a0 2024-01-13T11:49:41.214 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.222+0000 7f1f417fa700 1 --2- 172.21.15.59:0/3762578329 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1f2802fda0 0x7f1f28032260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:41.214 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.222+0000 7f1f417fa700 1 -- 172.21.15.59:0/3762578329 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f38041330 con 0x7f1f440704a0 2024-01-13T11:49:41.214 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.222+0000 7f1f43fff700 1 --2- 172.21.15.59:0/3762578329 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1f2802fda0 0x7f1f28032260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:41.214 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.222+0000 7f1f4b016700 1 -- 172.21.15.59:0/3762578329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f44059100 con 0x7f1f440704a0 2024-01-13T11:49:41.214 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.222+0000 7f1f43fff700 1 --2- 172.21.15.59:0/3762578329 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1f2802fda0 0x7f1f28032260 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1f300096a0 tx=0x7f1f30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:41.217 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.226+0000 7f1f417fa700 1 -- 172.21.15.59:0/3762578329 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f38016070 con 0x7f1f440704a0 2024-01-13T11:49:41.380 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.389+0000 7f1f4b016700 1 -- 172.21.15.59:0/3762578329 --> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f440ffc10 con 0x7f1f2802fda0 2024-01-13T11:49:41.447 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.456+0000 7f1f417fa700 1 -- 172.21.15.59:0/3762578329 <== mgr.14162 v2:172.21.15.59:6800/32893793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f1f440ffc10 con 0x7f1f2802fda0 2024-01-13T11:49:41.447 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: Self-signed certificate created 2024-01-13T11:49:41.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.458+0000 7f1f4b016700 1 -- 172.21.15.59:0/3762578329 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1f2802fda0 msgr2=0x7f1f28032260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:41.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.458+0000 7f1f4b016700 1 --2- 172.21.15.59:0/3762578329 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1f2802fda0 0x7f1f28032260 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1f300096a0 tx=0x7f1f30006b40).stop 2024-01-13T11:49:41.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.458+0000 7f1f4b016700 1 -- 172.21.15.59:0/3762578329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f440704a0 msgr2=0x7f1f440fc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:41.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.458+0000 7f1f4b016700 1 --2- 172.21.15.59:0/3762578329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f440704a0 0x7f1f440fc370 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f1f3800b190 tx=0x7f1f38006ee0).stop 2024-01-13T11:49:41.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.458+0000 7f1f4b016700 1 -- 172.21.15.59:0/3762578329 shutdown_connections 2024-01-13T11:49:41.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.458+0000 7f1f4b016700 1 --2- 172.21.15.59:0/3762578329 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1f2802fda0 0x7f1f28032260 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:41.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.458+0000 7f1f4b016700 1 --2- 172.21.15.59:0/3762578329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f440704a0 0x7f1f440fc370 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:41.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.458+0000 7f1f4b016700 1 -- 172.21.15.59:0/3762578329 >> 172.21.15.59:0/3762578329 conn(0x7f1f440f63b0 msgr2=0x7f1f440f7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:41.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.458+0000 7f1f4b016700 1 -- 172.21.15.59:0/3762578329 shutdown_connections 2024-01-13T11:49:41.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:41.459+0000 7f1f4b016700 1 -- 172.21.15.59:0/3762578329 wait complete. 2024-01-13T11:49:41.792 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:41 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:40.505262+0000 mgr.smithi059.hmyldj (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-13T11:49:41.792 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:41 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:40.505919+0000 mon.smithi059 (mon.0) 102 : cluster [DBG] mgrmap e15: smithi059.hmyldj(active, since 1.06321s) 2024-01-13T11:49:41.792 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:41 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:40.513201+0000 mgr.smithi059.hmyldj (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-13T11:49:41.792 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:41 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:41.456352+0000 mon.smithi059 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:41.792 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:41 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:41.458161+0000 mon.smithi059 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:41.815 INFO:teuthology.orchestra.run.smithi059.stderr:Creating initial admin user... 2024-01-13T11:49:41.815 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpefofk7vh:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-01-13T11:49:42.160 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.166+0000 7f6a64789700 1 Processor -- start 2024-01-13T11:49:42.160 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.166+0000 7f6a64789700 1 -- start start 2024-01-13T11:49:42.160 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.167+0000 7f6a64789700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a5c070390 0x7f6a5c100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:42.160 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.167+0000 7f6a64789700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c0fc770 con 0x7f6a5c070390 2024-01-13T11:49:42.160 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.167+0000 7f6a62525700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a5c070390 0x7f6a5c100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:42.160 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.167+0000 7f6a62525700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a5c070390 0x7f6a5c100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34848/0 (socket says 172.21.15.59:34848) 2024-01-13T11:49:42.161 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.167+0000 7f6a62525700 1 -- 172.21.15.59:0/3229510749 learned_addr learned my addr 172.21.15.59:0/3229510749 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:42.161 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.167+0000 7f6a62525700 1 -- 172.21.15.59:0/3229510749 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5c101270 con 0x7f6a5c070390 2024-01-13T11:49:42.161 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.167+0000 7f6a62525700 1 --2- 172.21.15.59:0/3229510749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a5c070390 0x7f6a5c100d80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f6a5800b5f0 tx=0x7f6a58006ee0).ready entity=mon.0 client_cookie=64c25675926cce10 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:42.161 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.167+0000 7f6a61523700 1 -- 172.21.15.59:0/3229510749 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a5800d600 con 0x7f6a5c070390 2024-01-13T11:49:42.161 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.167+0000 7f6a61523700 1 -- 172.21.15.59:0/3229510749 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6a5800dbd0 con 0x7f6a5c070390 2024-01-13T11:49:42.161 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.167+0000 7f6a61523700 1 -- 172.21.15.59:0/3229510749 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a58011c00 con 0x7f6a5c070390 2024-01-13T11:49:42.161 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.168+0000 7f6a64789700 1 -- 172.21.15.59:0/3229510749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a5c070390 msgr2=0x7f6a5c100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:42.161 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.168+0000 7f6a64789700 1 --2- 172.21.15.59:0/3229510749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a5c070390 0x7f6a5c100d80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f6a5800b5f0 tx=0x7f6a58006ee0).stop 2024-01-13T11:49:42.161 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.168+0000 7f6a64789700 1 -- 172.21.15.59:0/3229510749 shutdown_connections 2024-01-13T11:49:42.161 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.168+0000 7f6a64789700 1 --2- 172.21.15.59:0/3229510749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a5c070390 0x7f6a5c100d80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:42.161 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.168+0000 7f6a64789700 1 -- 172.21.15.59:0/3229510749 >> 172.21.15.59:0/3229510749 conn(0x7f6a5c0f63c0 msgr2=0x7f6a5c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:42.161 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.168+0000 7f6a64789700 1 -- 172.21.15.59:0/3229510749 shutdown_connections 2024-01-13T11:49:42.161 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.168+0000 7f6a64789700 1 -- 172.21.15.59:0/3229510749 wait complete. 2024-01-13T11:49:42.161 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.169+0000 7f6a64789700 1 Processor -- start 2024-01-13T11:49:42.162 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.169+0000 7f6a64789700 1 -- start start 2024-01-13T11:49:42.162 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.169+0000 7f6a64789700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a5c070390 0x7f6a5c10cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:42.162 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.169+0000 7f6a64789700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c10d0f0 con 0x7f6a5c070390 2024-01-13T11:49:42.162 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.169+0000 7f6a62525700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a5c070390 0x7f6a5c10cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:42.162 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.169+0000 7f6a62525700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a5c070390 0x7f6a5c10cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34852/0 (socket says 172.21.15.59:34852) 2024-01-13T11:49:42.162 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.169+0000 7f6a62525700 1 -- 172.21.15.59:0/3118132464 learned_addr learned my addr 172.21.15.59:0/3118132464 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:42.162 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.169+0000 7f6a62525700 1 -- 172.21.15.59:0/3118132464 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5800b200 con 0x7f6a5c070390 2024-01-13T11:49:42.162 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.169+0000 7f6a62525700 1 --2- 172.21.15.59:0/3118132464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a5c070390 0x7f6a5c10cc00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f6a5800b190 tx=0x7f6a58006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:42.162 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.170+0000 7f6a4effd700 1 -- 172.21.15.59:0/3118132464 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a5801b410 con 0x7f6a5c070390 2024-01-13T11:49:42.163 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.170+0000 7f6a64789700 1 -- 172.21.15.59:0/3118132464 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a5c10d2f0 con 0x7f6a5c070390 2024-01-13T11:49:42.164 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.170+0000 7f6a4effd700 1 -- 172.21.15.59:0/3118132464 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6a5801b9e0 con 0x7f6a5c070390 2024-01-13T11:49:42.164 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.170+0000 7f6a4effd700 1 -- 172.21.15.59:0/3118132464 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a5801a880 con 0x7f6a5c070390 2024-01-13T11:49:42.164 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.170+0000 7f6a64789700 1 -- 172.21.15.59:0/3118132464 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a5c10d700 con 0x7f6a5c070390 2024-01-13T11:49:42.164 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.170+0000 7f6a4effd700 1 -- 172.21.15.59:0/3118132464 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f6a5800f070 con 0x7f6a5c070390 2024-01-13T11:49:42.164 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.171+0000 7f6a4effd700 1 --2- 172.21.15.59:0/3118132464 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6a4402fd50 0x7f6a44032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:42.164 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.171+0000 7f6a4effd700 1 -- 172.21.15.59:0/3118132464 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6a580413a0 con 0x7f6a5c070390 2024-01-13T11:49:42.164 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.171+0000 7f6a61d24700 1 --2- 172.21.15.59:0/3118132464 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6a4402fd50 0x7f6a44032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:42.164 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.171+0000 7f6a61d24700 1 --2- 172.21.15.59:0/3118132464 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6a4402fd50 0x7f6a44032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f6a500096a0 tx=0x7f6a50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:42.164 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.171+0000 7f6a64789700 1 -- 172.21.15.59:0/3118132464 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a5c059100 con 0x7f6a5c070390 2024-01-13T11:49:42.166 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.175+0000 7f6a4effd700 1 -- 172.21.15.59:0/3118132464 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a58016070 con 0x7f6a5c070390 2024-01-13T11:49:42.171 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.181+0000 7f6a4effd700 1 -- 172.21.15.59:0/3118132464 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f6a5801bb50 con 0x7f6a5c070390 2024-01-13T11:49:42.339 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.348+0000 7f6a64789700 1 -- 172.21.15.59:0/3118132464 --> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f6a5c045ad0 con 0x7f6a4402fd50 2024-01-13T11:49:42.584 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.594+0000 7f6a4effd700 1 -- 172.21.15.59:0/3118132464 <== mgr.14162 v2:172.21.15.59:6800/32893793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f6a5c045ad0 con 0x7f6a4402fd50 2024-01-13T11:49:42.584 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$0018NY5FUibK3wyUvn2uAe8L/9KFZ3HAzxlQCRXXWHWoyqDagViHm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1705146582, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-13T11:49:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.596+0000 7f6a64789700 1 -- 172.21.15.59:0/3118132464 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6a4402fd50 msgr2=0x7f6a44032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.596+0000 7f6a64789700 1 --2- 172.21.15.59:0/3118132464 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6a4402fd50 0x7f6a44032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f6a500096a0 tx=0x7f6a50006b40).stop 2024-01-13T11:49:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.596+0000 7f6a64789700 1 -- 172.21.15.59:0/3118132464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a5c070390 msgr2=0x7f6a5c10cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.596+0000 7f6a64789700 1 --2- 172.21.15.59:0/3118132464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a5c070390 0x7f6a5c10cc00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f6a5800b190 tx=0x7f6a58006ee0).stop 2024-01-13T11:49:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.596+0000 7f6a64789700 1 -- 172.21.15.59:0/3118132464 shutdown_connections 2024-01-13T11:49:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.596+0000 7f6a64789700 1 --2- 172.21.15.59:0/3118132464 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6a4402fd50 0x7f6a44032210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.596+0000 7f6a64789700 1 --2- 172.21.15.59:0/3118132464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a5c070390 0x7f6a5c10cc00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.596+0000 7f6a64789700 1 -- 172.21.15.59:0/3118132464 >> 172.21.15.59:0/3118132464 conn(0x7f6a5c0f63c0 msgr2=0x7f6a5c0f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.596+0000 7f6a64789700 1 -- 172.21.15.59:0/3118132464 shutdown_connections 2024-01-13T11:49:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:42.596+0000 7f6a64789700 1 -- 172.21.15.59:0/3118132464 wait complete. 2024-01-13T11:49:42.889 INFO:teuthology.orchestra.run.smithi059.stderr:Fetching dashboard port number... 2024-01-13T11:49:42.889 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpghh114h5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp8ycx04qz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-01-13T11:49:43.272 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.280+0000 7fe099924700 1 Processor -- start 2024-01-13T11:49:43.272 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.280+0000 7fe099924700 1 -- start start 2024-01-13T11:49:43.272 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.280+0000 7fe099924700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe094072520 0x7fe094070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:43.272 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.280+0000 7fe099924700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe094072940 con 0x7fe094072520 2024-01-13T11:49:43.272 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.280+0000 7fe092ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe094072520 0x7fe094070ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:43.272 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.280+0000 7fe092ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe094072520 0x7fe094070ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34862/0 (socket says 172.21.15.59:34862) 2024-01-13T11:49:43.273 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.280+0000 7fe092ffd700 1 -- 172.21.15.59:0/1992019785 learned_addr learned my addr 172.21.15.59:0/1992019785 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:43.273 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.281+0000 7fe092ffd700 1 -- 172.21.15.59:0/1992019785 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe094070fb0 con 0x7fe094072520 2024-01-13T11:49:43.273 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.281+0000 7fe092ffd700 1 --2- 172.21.15.59:0/1992019785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe094072520 0x7fe094070ac0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fe08800b5f0 tx=0x7fe088006ee0).ready entity=mon.0 client_cookie=5f9e0f9985afc67 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:43.273 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.281+0000 7fe091ffb700 1 -- 172.21.15.59:0/1992019785 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe08800d600 con 0x7fe094072520 2024-01-13T11:49:43.273 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.281+0000 7fe091ffb700 1 -- 172.21.15.59:0/1992019785 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe08800dbd0 con 0x7fe094072520 2024-01-13T11:49:43.273 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.281+0000 7fe091ffb700 1 -- 172.21.15.59:0/1992019785 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe088011c00 con 0x7fe094072520 2024-01-13T11:49:43.277 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.282+0000 7fe099924700 1 -- 172.21.15.59:0/1992019785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe094072520 msgr2=0x7fe094070ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:43.277 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.282+0000 7fe099924700 1 --2- 172.21.15.59:0/1992019785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe094072520 0x7fe094070ac0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fe08800b5f0 tx=0x7fe088006ee0).stop 2024-01-13T11:49:43.277 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.282+0000 7fe099924700 1 -- 172.21.15.59:0/1992019785 shutdown_connections 2024-01-13T11:49:43.277 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.282+0000 7fe099924700 1 --2- 172.21.15.59:0/1992019785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe094072520 0x7fe094070ac0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:43.278 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.282+0000 7fe099924700 1 -- 172.21.15.59:0/1992019785 >> 172.21.15.59:0/1992019785 conn(0x7fe0940f74b0 msgr2=0x7fe0940f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:43.278 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.282+0000 7fe099924700 1 -- 172.21.15.59:0/1992019785 shutdown_connections 2024-01-13T11:49:43.278 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.282+0000 7fe099924700 1 -- 172.21.15.59:0/1992019785 wait complete. 2024-01-13T11:49:43.278 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.283+0000 7fe099924700 1 Processor -- start 2024-01-13T11:49:43.278 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.283+0000 7fe099924700 1 -- start start 2024-01-13T11:49:43.278 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.283+0000 7fe099924700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe094072520 0x7fe09410cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:43.278 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.283+0000 7fe099924700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09410d200 con 0x7fe094072520 2024-01-13T11:49:43.278 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.283+0000 7fe092ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe094072520 0x7fe09410cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:43.278 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.283+0000 7fe092ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe094072520 0x7fe09410cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34872/0 (socket says 172.21.15.59:34872) 2024-01-13T11:49:43.278 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.283+0000 7fe092ffd700 1 -- 172.21.15.59:0/1327925591 learned_addr learned my addr 172.21.15.59:0/1327925591 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:43.279 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.284+0000 7fe092ffd700 1 -- 172.21.15.59:0/1327925591 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08800b200 con 0x7fe094072520 2024-01-13T11:49:43.279 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.284+0000 7fe092ffd700 1 --2- 172.21.15.59:0/1327925591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe094072520 0x7fe09410cd10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fe08800b1d0 tx=0x7fe088006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:43.279 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.284+0000 7fe07b7fe700 1 -- 172.21.15.59:0/1327925591 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe088011a20 con 0x7fe094072520 2024-01-13T11:49:43.279 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.284+0000 7fe07b7fe700 1 -- 172.21.15.59:0/1327925591 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe08801b880 con 0x7fe094072520 2024-01-13T11:49:43.279 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.284+0000 7fe099924700 1 -- 172.21.15.59:0/1327925591 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe09410d400 con 0x7fe094072520 2024-01-13T11:49:43.279 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.284+0000 7fe07b7fe700 1 -- 172.21.15.59:0/1327925591 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe08801a700 con 0x7fe094072520 2024-01-13T11:49:43.279 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.284+0000 7fe099924700 1 -- 172.21.15.59:0/1327925591 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe09410d810 con 0x7fe094072520 2024-01-13T11:49:43.279 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.285+0000 7fe07b7fe700 1 -- 172.21.15.59:0/1327925591 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fe08800f070 con 0x7fe094072520 2024-01-13T11:49:43.279 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.285+0000 7fe07b7fe700 1 --2- 172.21.15.59:0/1327925591 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe08002ff00 0x7fe0800323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:43.280 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.285+0000 7fe07b7fe700 1 -- 172.21.15.59:0/1327925591 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe088041300 con 0x7fe094072520 2024-01-13T11:49:43.280 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.285+0000 7fe099924700 1 -- 172.21.15.59:0/1327925591 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe094045ad0 con 0x7fe094072520 2024-01-13T11:49:43.280 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.285+0000 7fe0927fc700 1 --2- 172.21.15.59:0/1327925591 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe08002ff00 0x7fe0800323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:43.280 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.285+0000 7fe0927fc700 1 --2- 172.21.15.59:0/1327925591 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe08002ff00 0x7fe0800323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe0840096a0 tx=0x7fe084006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:43.281 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.290+0000 7fe07b7fe700 1 -- 172.21.15.59:0/1327925591 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe088016070 con 0x7fe094072520 2024-01-13T11:49:43.437 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.447+0000 7fe099924700 1 -- 172.21.15.59:0/1327925591 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fe094101dd0 con 0x7fe094072520 2024-01-13T11:49:43.437 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.447+0000 7fe07b7fe700 1 -- 172.21.15.59:0/1327925591 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v9) v1 ==== 112+0+5 (secure 0 0 0) 0x7fe088019950 con 0x7fe094072520 2024-01-13T11:49:43.438 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 8443 2024-01-13T11:49:43.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.449+0000 7fe099924700 1 -- 172.21.15.59:0/1327925591 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe08002ff00 msgr2=0x7fe0800323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:43.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.449+0000 7fe099924700 1 --2- 172.21.15.59:0/1327925591 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe08002ff00 0x7fe0800323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe0840096a0 tx=0x7fe084006b40).stop 2024-01-13T11:49:43.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.449+0000 7fe099924700 1 -- 172.21.15.59:0/1327925591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe094072520 msgr2=0x7fe09410cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:43.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.449+0000 7fe099924700 1 --2- 172.21.15.59:0/1327925591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe094072520 0x7fe09410cd10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fe08800b1d0 tx=0x7fe088006ee0).stop 2024-01-13T11:49:43.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.449+0000 7fe099924700 1 -- 172.21.15.59:0/1327925591 shutdown_connections 2024-01-13T11:49:43.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.449+0000 7fe099924700 1 --2- 172.21.15.59:0/1327925591 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe08002ff00 0x7fe0800323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:43.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.449+0000 7fe099924700 1 --2- 172.21.15.59:0/1327925591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe094072520 0x7fe09410cd10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:43.441 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.449+0000 7fe099924700 1 -- 172.21.15.59:0/1327925591 >> 172.21.15.59:0/1327925591 conn(0x7fe0940f74b0 msgr2=0x7fe0940f9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:43.441 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.450+0000 7fe099924700 1 -- 172.21.15.59:0/1327925591 shutdown_connections 2024-01-13T11:49:43.441 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2024-01-13T11:49:43.450+0000 7fe099924700 1 -- 172.21.15.59:0/1327925591 wait complete. 2024-01-13T11:49:43.467 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:43 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:41.391886+0000 mgr.smithi059.hmyldj (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:49:43.467 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:43 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:42.184049+0000 mon.smithi059 (mon.0) 105 : cluster [DBG] mgrmap e16: smithi059.hmyldj(active, since 2s) 2024-01-13T11:49:43.468 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:43 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:42.595339+0000 mon.smithi059 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:43.773 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-13T11:49:43.782 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: disabled 2024-01-13T11:49:43.782 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: systemctl is-active firewalld.service 2024-01-13T11:49:43.789 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: inactive 2024-01-13T11:49:43.790 INFO:teuthology.orchestra.run.smithi059.stderr:firewalld.service is not enabled 2024-01-13T11:49:43.790 INFO:teuthology.orchestra.run.smithi059.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-13T11:49:43.793 INFO:teuthology.orchestra.run.smithi059.stderr:Ceph Dashboard is now available at: 2024-01-13T11:49:43.793 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-13T11:49:43.793 INFO:teuthology.orchestra.run.smithi059.stderr: URL: https://smithi059.front.sepia.ceph.com:8443/ 2024-01-13T11:49:43.793 INFO:teuthology.orchestra.run.smithi059.stderr: User: admin 2024-01-13T11:49:43.793 INFO:teuthology.orchestra.run.smithi059.stderr: Password: c9a4y3380p 2024-01-13T11:49:43.793 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-13T11:49:43.794 INFO:teuthology.orchestra.run.smithi059.stderr:You can access the Ceph CLI with: 2024-01-13T11:49:43.794 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-13T11:49:43.794 INFO:teuthology.orchestra.run.smithi059.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid a3276948-b209-11ee-95ac-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-13T11:49:43.794 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-13T11:49:43.794 INFO:teuthology.orchestra.run.smithi059.stderr:Please consider enabling telemetry to help improve Ceph: 2024-01-13T11:49:43.794 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-13T11:49:43.794 INFO:teuthology.orchestra.run.smithi059.stderr: ceph telemetry on 2024-01-13T11:49:43.794 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-13T11:49:43.794 INFO:teuthology.orchestra.run.smithi059.stderr:For more information see: 2024-01-13T11:49:43.794 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-13T11:49:43.794 INFO:teuthology.orchestra.run.smithi059.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-01-13T11:49:43.794 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-13T11:49:43.795 INFO:teuthology.orchestra.run.smithi059.stderr:Bootstrap complete. 2024-01-13T11:49:43.795 INFO:teuthology.orchestra.run.smithi059.stderr:Releasing lock 139994179828200 on /run/cephadm/a3276948-b209-11ee-95ac-87774f69a715.lock 2024-01-13T11:49:43.795 INFO:teuthology.orchestra.run.smithi059.stderr:Lock 139994179828200 released on /run/cephadm/a3276948-b209-11ee-95ac-87774f69a715.lock 2024-01-13T11:49:43.826 INFO:tasks.cephadm:Fetching config... 2024-01-13T11:49:43.826 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-13T11:49:43.826 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-13T11:49:43.885 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-13T11:49:43.885 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-13T11:49:43.885 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-13T11:49:43.943 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-13T11:49:43.943 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-13T11:49:43.943 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/mon.smithi059/keyring of=/dev/stdout 2024-01-13T11:49:44.011 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-13T11:49:44.012 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-13T11:49:44.012 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-13T11:49:44.071 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-13T11:49:44.071 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXLWI4l1k5ZexsEczQd4VVY2MeYhyIbfX57PRHiP8eeGzVCLZyujgbltA277plW7z5xppPUinicJyBGQXC2PkbStIzfjKazLbIHXgWh1GAw142UsyqrvZDAs5Mfd89DHt4u1IZjb7hgHIqfvGIy8vYPEXhtKjzcM+SQ3KLrh0lCqFwfRNB0JFSiLljRrxxSvamNwN3A4Uhl6Jq+dt/zx1juFT2btKeDuHuIpzzj7ztDIvCbOoI6YzznnnNx/qqxwv+2V4eJLBQ9EJWN0VTre/IJuSGc6RZxt1qmnv8+v469njUyXuoOariIH1rT2+vbYuswkmHIjkmkiB1ytzsRlbUYIfpTHLWR9dPkQq0Xh2861XXcWyMrV9+kdjYKuvraOQxzc+daTeg90aswiUvUbox9fgHvaR/SKPkJon0PKXyV+lYJ3Yh0QUj0usBYHXt6ZREhfwd6b/am3b2dBToOPh+uxj55D7e90ji2/rVybQ4OhMb3lzvyygMIrzAqQzA2eU= ceph-a3276948-b209-11ee-95ac-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-13T11:49:44.147 INFO:teuthology.orchestra.run.smithi059.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXLWI4l1k5ZexsEczQd4VVY2MeYhyIbfX57PRHiP8eeGzVCLZyujgbltA277plW7z5xppPUinicJyBGQXC2PkbStIzfjKazLbIHXgWh1GAw142UsyqrvZDAs5Mfd89DHt4u1IZjb7hgHIqfvGIy8vYPEXhtKjzcM+SQ3KLrh0lCqFwfRNB0JFSiLljRrxxSvamNwN3A4Uhl6Jq+dt/zx1juFT2btKeDuHuIpzzj7ztDIvCbOoI6YzznnnNx/qqxwv+2V4eJLBQ9EJWN0VTre/IJuSGc6RZxt1qmnv8+v469njUyXuoOariIH1rT2+vbYuswkmHIjkmkiB1ytzsRlbUYIfpTHLWR9dPkQq0Xh2861XXcWyMrV9+kdjYKuvraOQxzc+daTeg90aswiUvUbox9fgHvaR/SKPkJon0PKXyV+lYJ3Yh0QUj0usBYHXt6ZREhfwd6b/am3b2dBToOPh+uxj55D7e90ji2/rVybQ4OhMb3lzvyygMIrzAqQzA2eU= ceph-a3276948-b209-11ee-95ac-87774f69a715 2024-01-13T11:49:44.158 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXLWI4l1k5ZexsEczQd4VVY2MeYhyIbfX57PRHiP8eeGzVCLZyujgbltA277plW7z5xppPUinicJyBGQXC2PkbStIzfjKazLbIHXgWh1GAw142UsyqrvZDAs5Mfd89DHt4u1IZjb7hgHIqfvGIy8vYPEXhtKjzcM+SQ3KLrh0lCqFwfRNB0JFSiLljRrxxSvamNwN3A4Uhl6Jq+dt/zx1juFT2btKeDuHuIpzzj7ztDIvCbOoI6YzznnnNx/qqxwv+2V4eJLBQ9EJWN0VTre/IJuSGc6RZxt1qmnv8+v469njUyXuoOariIH1rT2+vbYuswkmHIjkmkiB1ytzsRlbUYIfpTHLWR9dPkQq0Xh2861XXcWyMrV9+kdjYKuvraOQxzc+daTeg90aswiUvUbox9fgHvaR/SKPkJon0PKXyV+lYJ3Yh0QUj0usBYHXt6ZREhfwd6b/am3b2dBToOPh+uxj55D7e90ji2/rVybQ4OhMb3lzvyygMIrzAqQzA2eU= ceph-a3276948-b209-11ee-95ac-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-13T11:49:44.203 INFO:teuthology.orchestra.run.smithi123.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXLWI4l1k5ZexsEczQd4VVY2MeYhyIbfX57PRHiP8eeGzVCLZyujgbltA277plW7z5xppPUinicJyBGQXC2PkbStIzfjKazLbIHXgWh1GAw142UsyqrvZDAs5Mfd89DHt4u1IZjb7hgHIqfvGIy8vYPEXhtKjzcM+SQ3KLrh0lCqFwfRNB0JFSiLljRrxxSvamNwN3A4Uhl6Jq+dt/zx1juFT2btKeDuHuIpzzj7ztDIvCbOoI6YzznnnNx/qqxwv+2V4eJLBQ9EJWN0VTre/IJuSGc6RZxt1qmnv8+v469njUyXuoOariIH1rT2+vbYuswkmHIjkmkiB1ytzsRlbUYIfpTHLWR9dPkQq0Xh2861XXcWyMrV9+kdjYKuvraOQxzc+daTeg90aswiUvUbox9fgHvaR/SKPkJon0PKXyV+lYJ3Yh0QUj0usBYHXt6ZREhfwd6b/am3b2dBToOPh+uxj55D7e90ji2/rVybQ4OhMb3lzvyygMIrzAqQzA2eU= ceph-a3276948-b209-11ee-95ac-87774f69a715 2024-01-13T11:49:44.215 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-13T11:49:44.388 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:44 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:42.350688+0000 mgr.smithi059.hmyldj (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:49:44.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:44 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:43.448776+0000 mon.smithi059 (mon.0) 107 : audit [DBG] from='client.? 172.21.15.59:0/1327925591' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-13T11:49:46.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.407+0000 7fc9c8bb8700 1 -- 172.21.15.59:0/3436631383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c40f6510 msgr2=0x7fc9c40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:46.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.407+0000 7fc9c8bb8700 1 --2- 172.21.15.59:0/3436631383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c40f6510 0x7fc9c40f6930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fc9b4009fb0 tx=0x7fc9b4009b30).stop 2024-01-13T11:49:46.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.407+0000 7fc9c8bb8700 1 -- 172.21.15.59:0/3436631383 shutdown_connections 2024-01-13T11:49:46.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.407+0000 7fc9c8bb8700 1 --2- 172.21.15.59:0/3436631383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c40f6510 0x7fc9c40f6930 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:46.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.407+0000 7fc9c8bb8700 1 -- 172.21.15.59:0/3436631383 >> 172.21.15.59:0/3436631383 conn(0x7fc9c40f1c70 msgr2=0x7fc9c40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:46.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.408+0000 7fc9c8bb8700 1 -- 172.21.15.59:0/3436631383 shutdown_connections 2024-01-13T11:49:46.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.408+0000 7fc9c8bb8700 1 -- 172.21.15.59:0/3436631383 wait complete. 2024-01-13T11:49:46.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.408+0000 7fc9c8bb8700 1 Processor -- start 2024-01-13T11:49:46.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.408+0000 7fc9c8bb8700 1 -- start start 2024-01-13T11:49:46.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.408+0000 7fc9c8bb8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c40f6510 0x7fc9c406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:46.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.408+0000 7fc9c8bb8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c4069e20 con 0x7fc9c40f6510 2024-01-13T11:49:46.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.408+0000 7fc9c259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c40f6510 0x7fc9c406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:46.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.408+0000 7fc9c259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c40f6510 0x7fc9c406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34896/0 (socket says 172.21.15.59:34896) 2024-01-13T11:49:46.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.408+0000 7fc9c259c700 1 -- 172.21.15.59:0/2883554790 learned_addr learned my addr 172.21.15.59:0/2883554790 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:49:46.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.409+0000 7fc9c259c700 1 -- 172.21.15.59:0/2883554790 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b4005040 con 0x7fc9c40f6510 2024-01-13T11:49:46.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.409+0000 7fc9c259c700 1 --2- 172.21.15.59:0/2883554790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c40f6510 0x7fc9c406b9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fc9b4000c00 tx=0x7fc9b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:46.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.409+0000 7fc9b2ffd700 1 -- 172.21.15.59:0/2883554790 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9b40095f0 con 0x7fc9c40f6510 2024-01-13T11:49:46.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.409+0000 7fc9b2ffd700 1 -- 172.21.15.59:0/2883554790 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc9b4007e30 con 0x7fc9c40f6510 2024-01-13T11:49:46.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.409+0000 7fc9c8bb8700 1 -- 172.21.15.59:0/2883554790 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c4069fc0 con 0x7fc9c40f6510 2024-01-13T11:49:46.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.409+0000 7fc9b2ffd700 1 -- 172.21.15.59:0/2883554790 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9b4014d20 con 0x7fc9c40f6510 2024-01-13T11:49:46.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.409+0000 7fc9c8bb8700 1 -- 172.21.15.59:0/2883554790 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c406a450 con 0x7fc9c40f6510 2024-01-13T11:49:46.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.410+0000 7fc9b2ffd700 1 -- 172.21.15.59:0/2883554790 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc9b4025070 con 0x7fc9c40f6510 2024-01-13T11:49:46.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.410+0000 7fc9c8bb8700 1 -- 172.21.15.59:0/2883554790 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c4059100 con 0x7fc9c40f6510 2024-01-13T11:49:46.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.410+0000 7fc9b2ffd700 1 --2- 172.21.15.59:0/2883554790 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc9ac02ff60 0x7fc9ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:49:46.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.410+0000 7fc9b2ffd700 1 -- 172.21.15.59:0/2883554790 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc9b40409a0 con 0x7fc9c40f6510 2024-01-13T11:49:46.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.410+0000 7fc9c1d9b700 1 --2- 172.21.15.59:0/2883554790 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc9ac02ff60 0x7fc9ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:49:46.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.410+0000 7fc9c1d9b700 1 --2- 172.21.15.59:0/2883554790 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc9ac02ff60 0x7fc9ac032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fc9b80096a0 tx=0x7fc9b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:49:46.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.414+0000 7fc9b2ffd700 1 -- 172.21.15.59:0/2883554790 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc9b400f4e0 con 0x7fc9c40f6510 2024-01-13T11:49:46.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.576+0000 7fc9c8bb8700 1 -- 172.21.15.59:0/2883554790 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fc9c40fbf50 con 0x7fc9c40f6510 2024-01-13T11:49:46.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.579+0000 7fc9b2ffd700 1 -- 172.21.15.59:0/2883554790 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v10)=0 v10) v1 ==== 125+0+0 (secure 0 0 0) 0x7fc9b400f300 con 0x7fc9c40f6510 2024-01-13T11:49:46.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.581+0000 7fc9c8bb8700 1 -- 172.21.15.59:0/2883554790 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc9ac02ff60 msgr2=0x7fc9ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:46.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.581+0000 7fc9c8bb8700 1 --2- 172.21.15.59:0/2883554790 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc9ac02ff60 0x7fc9ac032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fc9b80096a0 tx=0x7fc9b8006b40).stop 2024-01-13T11:49:46.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.581+0000 7fc9c8bb8700 1 -- 172.21.15.59:0/2883554790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c40f6510 msgr2=0x7fc9c406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:49:46.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.581+0000 7fc9c8bb8700 1 --2- 172.21.15.59:0/2883554790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c40f6510 0x7fc9c406b9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fc9b4000c00 tx=0x7fc9b400b040).stop 2024-01-13T11:49:46.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.581+0000 7fc9c8bb8700 1 -- 172.21.15.59:0/2883554790 shutdown_connections 2024-01-13T11:49:46.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.581+0000 7fc9c8bb8700 1 --2- 172.21.15.59:0/2883554790 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc9ac02ff60 0x7fc9ac032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:46.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.581+0000 7fc9c8bb8700 1 --2- 172.21.15.59:0/2883554790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c40f6510 0x7fc9c406b9a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:49:46.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.581+0000 7fc9c8bb8700 1 -- 172.21.15.59:0/2883554790 >> 172.21.15.59:0/2883554790 conn(0x7fc9c40f1c70 msgr2=0x7fc9c40f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:49:46.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.581+0000 7fc9c8bb8700 1 -- 172.21.15.59:0/2883554790 shutdown_connections 2024-01-13T11:49:46.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:49:46.581+0000 7fc9c8bb8700 1 -- 172.21.15.59:0/2883554790 wait complete. 2024-01-13T11:49:46.808 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:46 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:45.531975+0000 mon.smithi059 (mon.0) 108 : cluster [DBG] mgrmap e17: smithi059.hmyldj(active, since 6s) 2024-01-13T11:49:46.977 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi123 2024-01-13T11:49:46.977 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-01-13T11:49:46.977 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/etc/ceph/ceph.conf 2024-01-13T11:49:46.999 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-01-13T11:49:46.999 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-13T11:49:47.061 INFO:tasks.cephadm:Adding host smithi123 to orchestrator... 2024-01-13T11:49:47.062 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph orch host add smithi123 2024-01-13T11:49:47.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:47 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:46.581157+0000 mon.smithi059 (mon.0) 109 : audit [INF] from='client.? 172.21.15.59:0/2883554790' entity='client.admin' 2024-01-13T11:49:49.083 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:48 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:47.720900+0000 mon.smithi059 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:49.083 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:48 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:47.945078+0000 mon.smithi059 (mon.0) 111 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:49.083 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:48 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:48.612037+0000 mon.smithi059 (mon.0) 112 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:52.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:50.954513+0000 mon.smithi059 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:52.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:51.243030+0000 mon.smithi059 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:52.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:49:51.245237+0000 mon.smithi059 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' 2024-01-13T11:49:53.388 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:49:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cephadm 2024-01-13T11:49:51.251282+0000 mgr.smithi059.hmyldj (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi059 on smithi059 2024-01-13T11:50:00.889 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:00 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:49:59.502138+0000 mgr.smithi059.hmyldj (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:03.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:03 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:01.502537+0000 mgr.smithi059.hmyldj (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:05.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:05 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:03.502906+0000 mgr.smithi059.hmyldj (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:07.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:05.503223+0000 mgr.smithi059.hmyldj (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:09.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:09 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:07.503472+0000 mgr.smithi059.hmyldj (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:11.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:11 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:09.503705+0000 mgr.smithi059.hmyldj (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:13.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:13 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:11.504029+0000 mgr.smithi059.hmyldj (mgr.14162) 12 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:14.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.121+0000 7f4c1a900700 1 -- 172.21.15.123:0/126098114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c140f51a0 msgr2=0x7f4c140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:14.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.121+0000 7f4c1a900700 1 --2- 172.21.15.123:0/126098114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c140f51a0 0x7f4c140f55c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f4c04009fb0 tx=0x7f4c04009b30).stop 2024-01-13T11:50:14.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.121+0000 7f4c1a900700 1 -- 172.21.15.123:0/126098114 shutdown_connections 2024-01-13T11:50:14.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.121+0000 7f4c1a900700 1 --2- 172.21.15.123:0/126098114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c140f51a0 0x7f4c140f55c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:14.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.121+0000 7f4c1a900700 1 -- 172.21.15.123:0/126098114 >> 172.21.15.123:0/126098114 conn(0x7f4c140f09a0 msgr2=0x7f4c140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:14.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.121+0000 7f4c1a900700 1 -- 172.21.15.123:0/126098114 shutdown_connections 2024-01-13T11:50:14.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.121+0000 7f4c1a900700 1 -- 172.21.15.123:0/126098114 wait complete. 2024-01-13T11:50:14.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.121+0000 7f4c1a900700 1 Processor -- start 2024-01-13T11:50:14.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.121+0000 7f4c1a900700 1 -- start start 2024-01-13T11:50:14.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.122+0000 7f4c1a900700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c140f51a0 0x7f4c140fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:14.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.122+0000 7f4c1a900700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c140fb5a0 con 0x7f4c140f51a0 2024-01-13T11:50:14.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.122+0000 7f4c198fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c140f51a0 0x7f4c140fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:14.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.122+0000 7f4c198fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c140f51a0 0x7f4c140fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:48712/0 (socket says 172.21.15.123:48712) 2024-01-13T11:50:14.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.122+0000 7f4c198fe700 1 -- 172.21.15.123:0/2241456775 learned_addr learned my addr 172.21.15.123:0/2241456775 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:50:14.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.122+0000 7f4c198fe700 1 -- 172.21.15.123:0/2241456775 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c04005040 con 0x7f4c140f51a0 2024-01-13T11:50:14.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.122+0000 7f4c198fe700 1 --2- 172.21.15.123:0/2241456775 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c140f51a0 0x7f4c140fd120 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f4c04000c00 tx=0x7f4c0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:14.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.122+0000 7f4c0a7fc700 1 -- 172.21.15.123:0/2241456775 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c04009320 con 0x7f4c140f51a0 2024-01-13T11:50:14.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.122+0000 7f4c0a7fc700 1 -- 172.21.15.123:0/2241456775 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c04013ec0 con 0x7f4c140f51a0 2024-01-13T11:50:14.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.122+0000 7f4c1a900700 1 -- 172.21.15.123:0/2241456775 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c140fb740 con 0x7f4c140f51a0 2024-01-13T11:50:14.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.122+0000 7f4c0a7fc700 1 -- 172.21.15.123:0/2241456775 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c0401ecd0 con 0x7f4c140f51a0 2024-01-13T11:50:14.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.122+0000 7f4c1a900700 1 -- 172.21.15.123:0/2241456775 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c140fbbd0 con 0x7f4c140f51a0 2024-01-13T11:50:14.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.123+0000 7f4c0a7fc700 1 -- 172.21.15.123:0/2241456775 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c0401e420 con 0x7f4c140f51a0 2024-01-13T11:50:14.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.123+0000 7f4c1a900700 1 -- 172.21.15.123:0/2241456775 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c14059100 con 0x7f4c140f51a0 2024-01-13T11:50:14.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.123+0000 7f4c0a7fc700 1 --2- 172.21.15.123:0/2241456775 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f4c0002ff60 0x7f4c00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:14.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.123+0000 7f4c0a7fc700 1 -- 172.21.15.123:0/2241456775 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c04040e90 con 0x7f4c140f51a0 2024-01-13T11:50:14.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.124+0000 7f4c190fd700 1 --2- 172.21.15.123:0/2241456775 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f4c0002ff60 0x7f4c00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:14.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.124+0000 7f4c190fd700 1 --2- 172.21.15.123:0/2241456775 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f4c0002ff60 0x7f4c00032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f4c100096a0 tx=0x7f4c10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:14.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.128+0000 7f4c0a7fc700 1 -- 172.21.15.123:0/2241456775 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c0401e730 con 0x7f4c140f51a0 2024-01-13T11:50:14.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.302+0000 7f4c1a900700 1 -- 172.21.15.123:0/2241456775 --> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi123", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c140f27a0 con 0x7f4c0002ff60 2024-01-13T11:50:14.894 INFO:teuthology.orchestra.run.smithi123.stdout:Added host 'smithi123' 2024-01-13T11:50:14.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.903+0000 7f4c0a7fc700 1 -- 172.21.15.123:0/2241456775 <== mgr.14162 v2:172.21.15.59:6800/32893793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f4c140f27a0 con 0x7f4c0002ff60 2024-01-13T11:50:14.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.905+0000 7f4c1a900700 1 -- 172.21.15.123:0/2241456775 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f4c0002ff60 msgr2=0x7f4c00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:14.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.905+0000 7f4c1a900700 1 --2- 172.21.15.123:0/2241456775 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f4c0002ff60 0x7f4c00032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f4c100096a0 tx=0x7f4c10006b40).stop 2024-01-13T11:50:14.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.906+0000 7f4c1a900700 1 -- 172.21.15.123:0/2241456775 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c140f51a0 msgr2=0x7f4c140fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:14.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.906+0000 7f4c1a900700 1 --2- 172.21.15.123:0/2241456775 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c140f51a0 0x7f4c140fd120 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f4c04000c00 tx=0x7f4c0400b040).stop 2024-01-13T11:50:14.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.906+0000 7f4c1a900700 1 -- 172.21.15.123:0/2241456775 shutdown_connections 2024-01-13T11:50:14.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.906+0000 7f4c1a900700 1 --2- 172.21.15.123:0/2241456775 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f4c0002ff60 0x7f4c00032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:14.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.906+0000 7f4c1a900700 1 --2- 172.21.15.123:0/2241456775 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c140f51a0 0x7f4c140fd120 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:14.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.906+0000 7f4c1a900700 1 -- 172.21.15.123:0/2241456775 >> 172.21.15.123:0/2241456775 conn(0x7f4c140f09a0 msgr2=0x7f4c1405fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:14.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.906+0000 7f4c1a900700 1 -- 172.21.15.123:0/2241456775 shutdown_connections 2024-01-13T11:50:14.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:14.906+0000 7f4c1a900700 1 -- 172.21.15.123:0/2241456775 wait complete. 2024-01-13T11:50:15.219 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph orch host ls --format=json 2024-01-13T11:50:15.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:15 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:13.504360+0000 mgr.smithi059.hmyldj (mgr.14162) 13 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:15.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:15 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:14.903850+0000 mon.smithi059 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' 2024-01-13T11:50:16.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.271+0000 7f72bac52700 1 -- 172.21.15.123:0/3477022667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b40601c0 msgr2=0x7f72b40f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:16.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.271+0000 7f72bac52700 1 --2- 172.21.15.123:0/3477022667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b40601c0 0x7f72b40f91b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f72a8009fb0 tx=0x7f72a8009b30).stop 2024-01-13T11:50:16.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.271+0000 7f72bac52700 1 -- 172.21.15.123:0/3477022667 shutdown_connections 2024-01-13T11:50:16.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.271+0000 7f72bac52700 1 --2- 172.21.15.123:0/3477022667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b40601c0 0x7f72b40f91b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:16.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.271+0000 7f72bac52700 1 -- 172.21.15.123:0/3477022667 >> 172.21.15.123:0/3477022667 conn(0x7f72b40f0b70 msgr2=0x7f72b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:16.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.271+0000 7f72bac52700 1 -- 172.21.15.123:0/3477022667 shutdown_connections 2024-01-13T11:50:16.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.271+0000 7f72bac52700 1 -- 172.21.15.123:0/3477022667 wait complete. 2024-01-13T11:50:16.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.271+0000 7f72bac52700 1 Processor -- start 2024-01-13T11:50:16.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.271+0000 7f72bac52700 1 -- start start 2024-01-13T11:50:16.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.272+0000 7f72bac52700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b40601c0 0x7f72b41066f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:16.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.272+0000 7f72bac52700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b4106be0 con 0x7f72b40601c0 2024-01-13T11:50:16.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.272+0000 7f72b89ee700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b40601c0 0x7f72b41066f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:16.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.272+0000 7f72b89ee700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b40601c0 0x7f72b41066f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:48734/0 (socket says 172.21.15.123:48734) 2024-01-13T11:50:16.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.272+0000 7f72b89ee700 1 -- 172.21.15.123:0/2121095671 learned_addr learned my addr 172.21.15.123:0/2121095671 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:50:16.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.272+0000 7f72b89ee700 1 -- 172.21.15.123:0/2121095671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a8005040 con 0x7f72b40601c0 2024-01-13T11:50:16.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.272+0000 7f72b89ee700 1 --2- 172.21.15.123:0/2121095671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b40601c0 0x7f72b41066f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f72a8013510 tx=0x7f72a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:16.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.273+0000 7f72b17fa700 1 -- 172.21.15.123:0/2121095671 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72a8009320 con 0x7f72b40601c0 2024-01-13T11:50:16.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.273+0000 7f72bac52700 1 -- 172.21.15.123:0/2121095671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b4106de0 con 0x7f72b40601c0 2024-01-13T11:50:16.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.273+0000 7f72b17fa700 1 -- 172.21.15.123:0/2121095671 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72a8013f00 con 0x7f72b40601c0 2024-01-13T11:50:16.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.273+0000 7f72b17fa700 1 -- 172.21.15.123:0/2121095671 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72a8014dc0 con 0x7f72b40601c0 2024-01-13T11:50:16.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.273+0000 7f72bac52700 1 -- 172.21.15.123:0/2121095671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b4107270 con 0x7f72b40601c0 2024-01-13T11:50:16.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.273+0000 7f72b17fa700 1 -- 172.21.15.123:0/2121095671 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72a8014420 con 0x7f72b40601c0 2024-01-13T11:50:16.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.273+0000 7f72bac52700 1 -- 172.21.15.123:0/2121095671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b4045ad0 con 0x7f72b40601c0 2024-01-13T11:50:16.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.274+0000 7f72b17fa700 1 --2- 172.21.15.123:0/2121095671 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f72a402ff60 0x7f72a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:16.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.274+0000 7f72b17fa700 1 -- 172.21.15.123:0/2121095671 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72a8041110 con 0x7f72b40601c0 2024-01-13T11:50:16.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.274+0000 7f72b3fff700 1 --2- 172.21.15.123:0/2121095671 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f72a402ff60 0x7f72a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:16.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.274+0000 7f72b3fff700 1 --2- 172.21.15.123:0/2121095671 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f72a402ff60 0x7f72a4032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f72a00096a0 tx=0x7f72a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:16.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.279+0000 7f72b17fa700 1 -- 172.21.15.123:0/2121095671 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72a80095f0 con 0x7f72b40601c0 2024-01-13T11:50:16.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.448+0000 7f72bac52700 1 -- 172.21.15.123:0/2121095671 --> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f72b40f1da0 con 0x7f72a402ff60 2024-01-13T11:50:16.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.451+0000 7f72b17fa700 1 -- 172.21.15.123:0/2121095671 <== mgr.14162 v2:172.21.15.59:6800/32893793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7f72b40f1da0 con 0x7f72a402ff60 2024-01-13T11:50:16.443 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:50:16.443 INFO:teuthology.orchestra.run.smithi123.stdout:[{"addr": "172.21.15.59", "hostname": "smithi059", "labels": [], "status": ""}, {"addr": "smithi123", "hostname": "smithi123", "labels": [], "status": ""}] 2024-01-13T11:50:16.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.452+0000 7f72bac52700 1 -- 172.21.15.123:0/2121095671 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f72a402ff60 msgr2=0x7f72a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:16.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.452+0000 7f72bac52700 1 --2- 172.21.15.123:0/2121095671 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f72a402ff60 0x7f72a4032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f72a00096a0 tx=0x7f72a0006b40).stop 2024-01-13T11:50:16.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.452+0000 7f72bac52700 1 -- 172.21.15.123:0/2121095671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b40601c0 msgr2=0x7f72b41066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:16.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.452+0000 7f72bac52700 1 --2- 172.21.15.123:0/2121095671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b40601c0 0x7f72b41066f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f72a8013510 tx=0x7f72a800b040).stop 2024-01-13T11:50:16.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.452+0000 7f72bac52700 1 -- 172.21.15.123:0/2121095671 shutdown_connections 2024-01-13T11:50:16.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.452+0000 7f72bac52700 1 --2- 172.21.15.123:0/2121095671 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f72a402ff60 0x7f72a4032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:16.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.452+0000 7f72bac52700 1 --2- 172.21.15.123:0/2121095671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72b40601c0 0x7f72b41066f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:16.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.452+0000 7f72bac52700 1 -- 172.21.15.123:0/2121095671 >> 172.21.15.123:0/2121095671 conn(0x7f72b40f0b70 msgr2=0x7f72b40f1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:16.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.452+0000 7f72bac52700 1 -- 172.21.15.123:0/2121095671 shutdown_connections 2024-01-13T11:50:16.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:16.452+0000 7f72bac52700 1 -- 172.21.15.123:0/2121095671 wait complete. 2024-01-13T11:50:16.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:16 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:14.304116+0000 mgr.smithi059.hmyldj (mgr.14162) 14 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi123", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:50:16.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:16 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cephadm 2024-01-13T11:50:14.904275+0000 mgr.smithi059.hmyldj (mgr.14162) 15 : cephadm [INF] Added host smithi123 2024-01-13T11:50:16.822 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-13T11:50:16.822 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph osd crush tunables default 2024-01-13T11:50:17.542 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:17 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:15.504789+0000 mgr.smithi059.hmyldj (mgr.14162) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:18.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.081+0000 7f7b8aac6700 1 -- 172.21.15.59:0/2684526104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b840f5410 msgr2=0x7f7b840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:18.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.081+0000 7f7b8aac6700 1 --2- 172.21.15.59:0/2684526104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b840f5410 0x7f7b840f5830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f7b78009fb0 tx=0x7f7b78009b30).stop 2024-01-13T11:50:18.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.081+0000 7f7b8aac6700 1 -- 172.21.15.59:0/2684526104 shutdown_connections 2024-01-13T11:50:18.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.081+0000 7f7b8aac6700 1 --2- 172.21.15.59:0/2684526104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b840f5410 0x7f7b840f5830 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:18.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.081+0000 7f7b8aac6700 1 -- 172.21.15.59:0/2684526104 >> 172.21.15.59:0/2684526104 conn(0x7f7b840f0b70 msgr2=0x7f7b840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:18.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.081+0000 7f7b8aac6700 1 -- 172.21.15.59:0/2684526104 shutdown_connections 2024-01-13T11:50:18.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.081+0000 7f7b8aac6700 1 -- 172.21.15.59:0/2684526104 wait complete. 2024-01-13T11:50:18.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.081+0000 7f7b8aac6700 1 Processor -- start 2024-01-13T11:50:18.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.082+0000 7f7b8aac6700 1 -- start start 2024-01-13T11:50:18.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.082+0000 7f7b8aac6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b840f5410 0x7f7b84108930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:18.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.082+0000 7f7b8aac6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b84108e20 con 0x7f7b840f5410 2024-01-13T11:50:18.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.082+0000 7f7b88862700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b840f5410 0x7f7b84108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:18.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.082+0000 7f7b88862700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b840f5410 0x7f7b84108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47874/0 (socket says 172.21.15.59:47874) 2024-01-13T11:50:18.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.082+0000 7f7b88862700 1 -- 172.21.15.59:0/901551111 learned_addr learned my addr 172.21.15.59:0/901551111 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-13T11:50:18.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.082+0000 7f7b88862700 1 -- 172.21.15.59:0/901551111 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b78005040 con 0x7f7b840f5410 2024-01-13T11:50:18.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.082+0000 7f7b88862700 1 --2- 172.21.15.59:0/901551111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b840f5410 0x7f7b84108930 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f7b78013510 tx=0x7f7b7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:18.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.082+0000 7f7b817fa700 1 -- 172.21.15.59:0/901551111 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b78009320 con 0x7f7b840f5410 2024-01-13T11:50:18.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.083+0000 7f7b8aac6700 1 -- 172.21.15.59:0/901551111 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b84109020 con 0x7f7b840f5410 2024-01-13T11:50:18.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.083+0000 7f7b817fa700 1 -- 172.21.15.59:0/901551111 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b78013f00 con 0x7f7b840f5410 2024-01-13T11:50:18.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.083+0000 7f7b817fa700 1 -- 172.21.15.59:0/901551111 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b78014d90 con 0x7f7b840f5410 2024-01-13T11:50:18.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.083+0000 7f7b8aac6700 1 -- 172.21.15.59:0/901551111 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b841094b0 con 0x7f7b840f5410 2024-01-13T11:50:18.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.083+0000 7f7b817fa700 1 -- 172.21.15.59:0/901551111 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b780263a0 con 0x7f7b840f5410 2024-01-13T11:50:18.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.084+0000 7f7b8aac6700 1 -- 172.21.15.59:0/901551111 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b84045ad0 con 0x7f7b840f5410 2024-01-13T11:50:18.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.084+0000 7f7b817fa700 1 --2- 172.21.15.59:0/901551111 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7b7402ff60 0x7f7b74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:18.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.084+0000 7f7b817fa700 1 -- 172.21.15.59:0/901551111 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b78041320 con 0x7f7b840f5410 2024-01-13T11:50:18.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.084+0000 7f7b83fff700 1 --2- 172.21.15.59:0/901551111 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7b7402ff60 0x7f7b74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:18.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.084+0000 7f7b83fff700 1 --2- 172.21.15.59:0/901551111 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7b7402ff60 0x7f7b74032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f7b700096a0 tx=0x7f7b70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:18.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.088+0000 7f7b817fa700 1 -- 172.21.15.59:0/901551111 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b780126f0 con 0x7f7b840f5410 2024-01-13T11:50:18.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:18.256+0000 7f7b8aac6700 1 -- 172.21.15.59:0/901551111 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f7b8410a360 con 0x7f7b840f5410 2024-01-13T11:50:18.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:18 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:16.450904+0000 mgr.smithi059.hmyldj (mgr.14162) 17 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-13T11:50:19.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:19.210+0000 7f7b817fa700 1 -- 172.21.15.59:0/901551111 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f7b78012330 con 0x7f7b840f5410 2024-01-13T11:50:19.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:19.212+0000 7f7b8aac6700 1 -- 172.21.15.59:0/901551111 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7b7402ff60 msgr2=0x7f7b74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:19.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:19.212+0000 7f7b8aac6700 1 --2- 172.21.15.59:0/901551111 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7b7402ff60 0x7f7b74032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f7b700096a0 tx=0x7f7b70006b40).stop 2024-01-13T11:50:19.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:19.212+0000 7f7b8aac6700 1 -- 172.21.15.59:0/901551111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b840f5410 msgr2=0x7f7b84108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:19.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:19.212+0000 7f7b8aac6700 1 --2- 172.21.15.59:0/901551111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b840f5410 0x7f7b84108930 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f7b78013510 tx=0x7f7b7800b040).stop 2024-01-13T11:50:19.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:19.212+0000 7f7b8aac6700 1 -- 172.21.15.59:0/901551111 shutdown_connections 2024-01-13T11:50:19.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:19.212+0000 7f7b8aac6700 1 --2- 172.21.15.59:0/901551111 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7b7402ff60 0x7f7b74032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:19.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:19.212+0000 7f7b8aac6700 1 --2- 172.21.15.59:0/901551111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b840f5410 0x7f7b84108930 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:19.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:19.212+0000 7f7b8aac6700 1 -- 172.21.15.59:0/901551111 >> 172.21.15.59:0/901551111 conn(0x7f7b840f0b70 msgr2=0x7f7b840f2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:19.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:19.213+0000 7f7b8aac6700 1 -- 172.21.15.59:0/901551111 shutdown_connections 2024-01-13T11:50:19.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-13T11:50:19.213+0000 7f7b8aac6700 1 -- 172.21.15.59:0/901551111 wait complete. 2024-01-13T11:50:19.203 INFO:teuthology.orchestra.run.smithi059.stderr:adjusted tunables profile to default 2024-01-13T11:50:19.545 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:19 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:17.505362+0000 mgr.smithi059.hmyldj (mgr.14162) 18 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:19.546 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:19 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:18.256658+0000 mon.smithi059 (mon.0) 117 : audit [INF] from='client.? 172.21.15.59:0/901551111' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-13T11:50:19.601 INFO:tasks.cephadm:Adding mon.smithi059 on smithi059 2024-01-13T11:50:19.601 INFO:tasks.cephadm:Adding mon.smithi123 on smithi123 2024-01-13T11:50:19.602 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph orch apply mon '2;smithi059:172.21.15.59=smithi059;smithi123:172.21.15.123=smithi123' 2024-01-13T11:50:20.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:20 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:19.210707+0000 mon.smithi059 (mon.0) 118 : audit [INF] from='client.? 172.21.15.59:0/901551111' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-13T11:50:20.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:20 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:19.210762+0000 mon.smithi059 (mon.0) 119 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-13T11:50:20.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.712+0000 7ffb2dca9700 1 -- 172.21.15.123:0/2775451229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb28060280 msgr2=0x7ffb280f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:20.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.712+0000 7ffb2dca9700 1 --2- 172.21.15.123:0/2775451229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb28060280 0x7ffb280f90f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7ffb1c009fb0 tx=0x7ffb1c009b30).stop 2024-01-13T11:50:20.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.712+0000 7ffb2dca9700 1 -- 172.21.15.123:0/2775451229 shutdown_connections 2024-01-13T11:50:20.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.712+0000 7ffb2dca9700 1 --2- 172.21.15.123:0/2775451229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb28060280 0x7ffb280f90f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:20.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.712+0000 7ffb2dca9700 1 -- 172.21.15.123:0/2775451229 >> 172.21.15.123:0/2775451229 conn(0x7ffb280f0b70 msgr2=0x7ffb280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:20.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.712+0000 7ffb2dca9700 1 -- 172.21.15.123:0/2775451229 shutdown_connections 2024-01-13T11:50:20.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.712+0000 7ffb2dca9700 1 -- 172.21.15.123:0/2775451229 wait complete. 2024-01-13T11:50:20.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.713+0000 7ffb2dca9700 1 Processor -- start 2024-01-13T11:50:20.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.713+0000 7ffb2dca9700 1 -- start start 2024-01-13T11:50:20.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.713+0000 7ffb2dca9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb28060280 0x7ffb281066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:20.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.713+0000 7ffb2dca9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb28106ba0 con 0x7ffb28060280 2024-01-13T11:50:20.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.713+0000 7ffb277fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb28060280 0x7ffb281066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:20.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.713+0000 7ffb277fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb28060280 0x7ffb281066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:52602/0 (socket says 172.21.15.123:52602) 2024-01-13T11:50:20.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.713+0000 7ffb277fe700 1 -- 172.21.15.123:0/1222117333 learned_addr learned my addr 172.21.15.123:0/1222117333 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:50:20.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.714+0000 7ffb277fe700 1 -- 172.21.15.123:0/1222117333 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1c005040 con 0x7ffb28060280 2024-01-13T11:50:20.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.714+0000 7ffb277fe700 1 --2- 172.21.15.123:0/1222117333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb28060280 0x7ffb281066b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7ffb1c013510 tx=0x7ffb1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:20.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.714+0000 7ffb0ffff700 1 -- 172.21.15.123:0/1222117333 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb1c009320 con 0x7ffb28060280 2024-01-13T11:50:20.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.714+0000 7ffb2dca9700 1 -- 172.21.15.123:0/1222117333 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb28106da0 con 0x7ffb28060280 2024-01-13T11:50:20.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.714+0000 7ffb0ffff700 1 -- 172.21.15.123:0/1222117333 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb1c013f00 con 0x7ffb28060280 2024-01-13T11:50:20.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.714+0000 7ffb0ffff700 1 -- 172.21.15.123:0/1222117333 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb1c014dc0 con 0x7ffb28060280 2024-01-13T11:50:20.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.714+0000 7ffb2dca9700 1 -- 172.21.15.123:0/1222117333 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb28107230 con 0x7ffb28060280 2024-01-13T11:50:20.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.715+0000 7ffb0ffff700 1 -- 172.21.15.123:0/1222117333 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffb1c0095f0 con 0x7ffb28060280 2024-01-13T11:50:20.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.715+0000 7ffb0ffff700 1 --2- 172.21.15.123:0/1222117333 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffb1002ffb0 0x7ffb10032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:20.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.715+0000 7ffb2dca9700 1 -- 172.21.15.123:0/1222117333 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb28045ad0 con 0x7ffb28060280 2024-01-13T11:50:20.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.715+0000 7ffb0ffff700 1 -- 172.21.15.123:0/1222117333 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffb1c040ff0 con 0x7ffb28060280 2024-01-13T11:50:20.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.715+0000 7ffb26ffd700 1 --2- 172.21.15.123:0/1222117333 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffb1002ffb0 0x7ffb10032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:20.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.715+0000 7ffb26ffd700 1 --2- 172.21.15.123:0/1222117333 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffb1002ffb0 0x7ffb10032470 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7ffb180096a0 tx=0x7ffb18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:20.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.720+0000 7ffb0ffff700 1 -- 172.21.15.123:0/1222117333 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb1c014420 con 0x7ffb28060280 2024-01-13T11:50:20.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.895+0000 7ffb2dca9700 1 -- 172.21.15.123:0/1222117333 --> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi059:172.21.15.59=smithi059;smithi123:172.21.15.123=smithi123", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb280f1d20 con 0x7ffb1002ffb0 2024-01-13T11:50:20.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.901+0000 7ffb0ffff700 1 -- 172.21.15.123:0/1222117333 <== mgr.14162 v2:172.21.15.59:6800/32893793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ffb280f1d20 con 0x7ffb1002ffb0 2024-01-13T11:50:20.890 INFO:teuthology.orchestra.run.smithi123.stdout:Scheduled mon update... 2024-01-13T11:50:20.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.902+0000 7ffb2dca9700 1 -- 172.21.15.123:0/1222117333 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffb1002ffb0 msgr2=0x7ffb10032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:20.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.902+0000 7ffb2dca9700 1 --2- 172.21.15.123:0/1222117333 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffb1002ffb0 0x7ffb10032470 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7ffb180096a0 tx=0x7ffb18006b40).stop 2024-01-13T11:50:20.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.902+0000 7ffb2dca9700 1 -- 172.21.15.123:0/1222117333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb28060280 msgr2=0x7ffb281066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:20.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.902+0000 7ffb2dca9700 1 --2- 172.21.15.123:0/1222117333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb28060280 0x7ffb281066b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7ffb1c013510 tx=0x7ffb1c00b040).stop 2024-01-13T11:50:20.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.902+0000 7ffb2dca9700 1 -- 172.21.15.123:0/1222117333 shutdown_connections 2024-01-13T11:50:20.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.902+0000 7ffb2dca9700 1 --2- 172.21.15.123:0/1222117333 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffb1002ffb0 0x7ffb10032470 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:20.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.902+0000 7ffb2dca9700 1 --2- 172.21.15.123:0/1222117333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb28060280 0x7ffb281066b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:20.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.902+0000 7ffb2dca9700 1 -- 172.21.15.123:0/1222117333 >> 172.21.15.123:0/1222117333 conn(0x7ffb280f0b70 msgr2=0x7ffb280f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:20.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.903+0000 7ffb2dca9700 1 -- 172.21.15.123:0/1222117333 shutdown_connections 2024-01-13T11:50:20.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:20.903+0000 7ffb2dca9700 1 -- 172.21.15.123:0/1222117333 wait complete. 2024-01-13T11:50:21.253 DEBUG:teuthology.orchestra.run.smithi123:mon.smithi123> sudo journalctl -f -n 0 -u ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi123.service 2024-01-13T11:50:21.255 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:50:21.256 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:50:21.284 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:-- Logs begin at Sat 2024-01-13 11:37:27 UTC. -- 2024-01-13T11:50:21.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:21 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:19.505692+0000 mgr.smithi059.hmyldj (mgr.14162) 19 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:21.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:21 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:20.901635+0000 mon.smithi059 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' 2024-01-13T11:50:22.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.348+0000 7fbd1756b700 1 -- 172.21.15.123:0/1893745642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100f5410 msgr2=0x7fbd100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:22.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.348+0000 7fbd1756b700 1 --2- 172.21.15.123:0/1893745642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100f5410 0x7fbd100f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fbd04009fb0 tx=0x7fbd04009b30).stop 2024-01-13T11:50:22.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.348+0000 7fbd1756b700 1 -- 172.21.15.123:0/1893745642 shutdown_connections 2024-01-13T11:50:22.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.348+0000 7fbd1756b700 1 --2- 172.21.15.123:0/1893745642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100f5410 0x7fbd100f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:22.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.348+0000 7fbd1756b700 1 -- 172.21.15.123:0/1893745642 >> 172.21.15.123:0/1893745642 conn(0x7fbd100f0b70 msgr2=0x7fbd100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:22.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.348+0000 7fbd1756b700 1 -- 172.21.15.123:0/1893745642 shutdown_connections 2024-01-13T11:50:22.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.348+0000 7fbd1756b700 1 -- 172.21.15.123:0/1893745642 wait complete. 2024-01-13T11:50:22.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.349+0000 7fbd1756b700 1 Processor -- start 2024-01-13T11:50:22.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.349+0000 7fbd1756b700 1 -- start start 2024-01-13T11:50:22.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.349+0000 7fbd1756b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100f5410 0x7fbd101093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:22.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.349+0000 7fbd1756b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd10060280 con 0x7fbd100f5410 2024-01-13T11:50:22.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.349+0000 7fbd15307700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100f5410 0x7fbd101093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:22.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.349+0000 7fbd15307700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100f5410 0x7fbd101093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:52614/0 (socket says 172.21.15.123:52614) 2024-01-13T11:50:22.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.349+0000 7fbd15307700 1 -- 172.21.15.123:0/1384633502 learned_addr learned my addr 172.21.15.123:0/1384633502 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:50:22.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.349+0000 7fbd15307700 1 -- 172.21.15.123:0/1384633502 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd04005040 con 0x7fbd100f5410 2024-01-13T11:50:22.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.350+0000 7fbd15307700 1 --2- 172.21.15.123:0/1384633502 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100f5410 0x7fbd101093f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fbd04013510 tx=0x7fbd0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:22.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.350+0000 7fbd01ffb700 1 -- 172.21.15.123:0/1384633502 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd04009320 con 0x7fbd100f5410 2024-01-13T11:50:22.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.350+0000 7fbd1756b700 1 -- 172.21.15.123:0/1384633502 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd10060480 con 0x7fbd100f5410 2024-01-13T11:50:22.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.350+0000 7fbd01ffb700 1 -- 172.21.15.123:0/1384633502 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd04013f00 con 0x7fbd100f5410 2024-01-13T11:50:22.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.350+0000 7fbd01ffb700 1 -- 172.21.15.123:0/1384633502 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd04014d90 con 0x7fbd100f5410 2024-01-13T11:50:22.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.350+0000 7fbd1756b700 1 -- 172.21.15.123:0/1384633502 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd10109b20 con 0x7fbd100f5410 2024-01-13T11:50:22.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.351+0000 7fbd01ffb700 1 -- 172.21.15.123:0/1384633502 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd04014420 con 0x7fbd100f5410 2024-01-13T11:50:22.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.351+0000 7fbd1756b700 1 -- 172.21.15.123:0/1384633502 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd10045ad0 con 0x7fbd100f5410 2024-01-13T11:50:22.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.351+0000 7fbd01ffb700 1 --2- 172.21.15.123:0/1384633502 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbcfc02ff60 0x7fbcfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:22.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.351+0000 7fbd01ffb700 1 -- 172.21.15.123:0/1384633502 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd04041000 con 0x7fbd100f5410 2024-01-13T11:50:22.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.351+0000 7fbd14b06700 1 --2- 172.21.15.123:0/1384633502 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbcfc02ff60 0x7fbcfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:22.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.352+0000 7fbd14b06700 1 --2- 172.21.15.123:0/1384633502 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbcfc02ff60 0x7fbcfc032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fbd0c0096a0 tx=0x7fbd0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:22.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.356+0000 7fbd01ffb700 1 -- 172.21.15.123:0/1384633502 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd040095f0 con 0x7fbd100f5410 2024-01-13T11:50:22.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.561+0000 7fbd1756b700 1 -- 172.21.15.123:0/1384633502 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd1010a340 con 0x7fbd100f5410 2024-01-13T11:50:22.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.562+0000 7fbd01ffb700 1 -- 172.21.15.123:0/1384633502 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd0401db70 con 0x7fbd100f5410 2024-01-13T11:50:22.552 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:50:22.552 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:50:22.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.563+0000 7fbd1756b700 1 -- 172.21.15.123:0/1384633502 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbcfc02ff60 msgr2=0x7fbcfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:22.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.563+0000 7fbd1756b700 1 --2- 172.21.15.123:0/1384633502 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbcfc02ff60 0x7fbcfc032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fbd0c0096a0 tx=0x7fbd0c006b40).stop 2024-01-13T11:50:22.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.563+0000 7fbd1756b700 1 -- 172.21.15.123:0/1384633502 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100f5410 msgr2=0x7fbd101093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:22.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.563+0000 7fbd1756b700 1 --2- 172.21.15.123:0/1384633502 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100f5410 0x7fbd101093f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fbd04013510 tx=0x7fbd0400b040).stop 2024-01-13T11:50:22.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.563+0000 7fbd1756b700 1 -- 172.21.15.123:0/1384633502 shutdown_connections 2024-01-13T11:50:22.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.563+0000 7fbd1756b700 1 --2- 172.21.15.123:0/1384633502 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbcfc02ff60 0x7fbcfc032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:22.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.563+0000 7fbd1756b700 1 --2- 172.21.15.123:0/1384633502 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100f5410 0x7fbd101093f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:22.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.563+0000 7fbd1756b700 1 -- 172.21.15.123:0/1384633502 >> 172.21.15.123:0/1384633502 conn(0x7fbd100f0b70 msgr2=0x7fbd100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:22.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.563+0000 7fbd1756b700 1 -- 172.21.15.123:0/1384633502 shutdown_connections 2024-01-13T11:50:22.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:22.563+0000 7fbd1756b700 1 -- 172.21.15.123:0/1384633502 wait complete. 2024-01-13T11:50:22.554 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:50:22.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:22 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:20.897677+0000 mgr.smithi059.hmyldj (mgr.14162) 20 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi059:172.21.15.59=smithi059;smithi123:172.21.15.123=smithi123", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:50:22.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:22 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cephadm 2024-01-13T11:50:20.899085+0000 mgr.smithi059.hmyldj (mgr.14162) 21 : cephadm [INF] Saving service mon spec with placement smithi059:172.21.15.59=smithi059;smithi123:172.21.15.123=smithi123;count:2 2024-01-13T11:50:23.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:23 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:21.506043+0000 mgr.smithi059.hmyldj (mgr.14162) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:23.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:23 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:22.562868+0000 mon.smithi059 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.123:0/1384633502' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:50:23.955 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:50:23.956 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:50:25.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.059+0000 7fe35d42a700 1 -- 172.21.15.123:0/409337221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3580f51a0 msgr2=0x7fe3580f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:25.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.059+0000 7fe35d42a700 1 --2- 172.21.15.123:0/409337221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3580f51a0 0x7fe3580f55c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fe348009fb0 tx=0x7fe348009b30).stop 2024-01-13T11:50:25.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.059+0000 7fe35d42a700 1 -- 172.21.15.123:0/409337221 shutdown_connections 2024-01-13T11:50:25.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.059+0000 7fe35d42a700 1 --2- 172.21.15.123:0/409337221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3580f51a0 0x7fe3580f55c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:25.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.059+0000 7fe35d42a700 1 -- 172.21.15.123:0/409337221 >> 172.21.15.123:0/409337221 conn(0x7fe3580f09a0 msgr2=0x7fe3580f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:25.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.059+0000 7fe35d42a700 1 -- 172.21.15.123:0/409337221 shutdown_connections 2024-01-13T11:50:25.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.059+0000 7fe35d42a700 1 -- 172.21.15.123:0/409337221 wait complete. 2024-01-13T11:50:25.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.059+0000 7fe35d42a700 1 Processor -- start 2024-01-13T11:50:25.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.059+0000 7fe35d42a700 1 -- start start 2024-01-13T11:50:25.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.060+0000 7fe35d42a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3580f51a0 0x7fe3581091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:25.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.060+0000 7fe35d42a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe358060240 con 0x7fe3580f51a0 2024-01-13T11:50:25.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.060+0000 7fe357fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3580f51a0 0x7fe3581091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:25.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.060+0000 7fe357fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3580f51a0 0x7fe3581091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:52630/0 (socket says 172.21.15.123:52630) 2024-01-13T11:50:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.060+0000 7fe357fff700 1 -- 172.21.15.123:0/3062407662 learned_addr learned my addr 172.21.15.123:0/3062407662 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:50:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.060+0000 7fe357fff700 1 -- 172.21.15.123:0/3062407662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe348005040 con 0x7fe3580f51a0 2024-01-13T11:50:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.060+0000 7fe357fff700 1 --2- 172.21.15.123:0/3062407662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3580f51a0 0x7fe3581091b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fe348013510 tx=0x7fe34800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.060+0000 7fe354ff9700 1 -- 172.21.15.123:0/3062407662 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe348009320 con 0x7fe3580f51a0 2024-01-13T11:50:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.060+0000 7fe35d42a700 1 -- 172.21.15.123:0/3062407662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe358060440 con 0x7fe3580f51a0 2024-01-13T11:50:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.060+0000 7fe354ff9700 1 -- 172.21.15.123:0/3062407662 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe348013f00 con 0x7fe3580f51a0 2024-01-13T11:50:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.060+0000 7fe354ff9700 1 -- 172.21.15.123:0/3062407662 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe348014dc0 con 0x7fe3580f51a0 2024-01-13T11:50:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.060+0000 7fe35d42a700 1 -- 172.21.15.123:0/3062407662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe358109860 con 0x7fe3580f51a0 2024-01-13T11:50:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.061+0000 7fe354ff9700 1 -- 172.21.15.123:0/3062407662 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3480095f0 con 0x7fe3580f51a0 2024-01-13T11:50:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.061+0000 7fe35d42a700 1 -- 172.21.15.123:0/3062407662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe358045ad0 con 0x7fe3580f51a0 2024-01-13T11:50:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.061+0000 7fe354ff9700 1 --2- 172.21.15.123:0/3062407662 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe34002ff60 0x7fe340032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.061+0000 7fe354ff9700 1 -- 172.21.15.123:0/3062407662 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe3480410c0 con 0x7fe3580f51a0 2024-01-13T11:50:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.061+0000 7fe3577fe700 1 --2- 172.21.15.123:0/3062407662 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe34002ff60 0x7fe340032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:25.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.062+0000 7fe3577fe700 1 --2- 172.21.15.123:0/3062407662 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe34002ff60 0x7fe340032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fe34c0096a0 tx=0x7fe34c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:25.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.066+0000 7fe354ff9700 1 -- 172.21.15.123:0/3062407662 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe348014420 con 0x7fe3580f51a0 2024-01-13T11:50:25.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.277+0000 7fe35d42a700 1 -- 172.21.15.123:0/3062407662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe35810a080 con 0x7fe3580f51a0 2024-01-13T11:50:25.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.277+0000 7fe354ff9700 1 -- 172.21.15.123:0/3062407662 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe348010230 con 0x7fe3580f51a0 2024-01-13T11:50:25.266 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:50:25.266 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:50:25.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.279+0000 7fe35d42a700 1 -- 172.21.15.123:0/3062407662 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe34002ff60 msgr2=0x7fe340032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:25.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.279+0000 7fe35d42a700 1 --2- 172.21.15.123:0/3062407662 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe34002ff60 0x7fe340032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fe34c0096a0 tx=0x7fe34c006b40).stop 2024-01-13T11:50:25.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.279+0000 7fe35d42a700 1 -- 172.21.15.123:0/3062407662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3580f51a0 msgr2=0x7fe3581091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:25.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.279+0000 7fe35d42a700 1 --2- 172.21.15.123:0/3062407662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3580f51a0 0x7fe3581091b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fe348013510 tx=0x7fe34800b040).stop 2024-01-13T11:50:25.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.279+0000 7fe35d42a700 1 -- 172.21.15.123:0/3062407662 shutdown_connections 2024-01-13T11:50:25.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.279+0000 7fe35d42a700 1 --2- 172.21.15.123:0/3062407662 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe34002ff60 0x7fe340032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:25.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.279+0000 7fe35d42a700 1 --2- 172.21.15.123:0/3062407662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3580f51a0 0x7fe3581091b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:25.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.279+0000 7fe35d42a700 1 -- 172.21.15.123:0/3062407662 >> 172.21.15.123:0/3062407662 conn(0x7fe3580f09a0 msgr2=0x7fe3580f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:25.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.279+0000 7fe35d42a700 1 -- 172.21.15.123:0/3062407662 shutdown_connections 2024-01-13T11:50:25.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:25.279+0000 7fe35d42a700 1 -- 172.21.15.123:0/3062407662 wait complete. 2024-01-13T11:50:25.269 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:50:25.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:25 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:23.506458+0000 mgr.smithi059.hmyldj (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:26.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:26 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:25.277618+0000 mon.smithi059 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.123:0/3062407662' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:50:26.640 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:50:26.641 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:50:27.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:27 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:25.506748+0000 mgr.smithi059.hmyldj (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:27.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.776+0000 7f34b0f18700 1 -- 172.21.15.123:0/822719874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34ac060280 msgr2=0x7f34ac0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:27.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.776+0000 7f34b0f18700 1 --2- 172.21.15.123:0/822719874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34ac060280 0x7f34ac0f90f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f34a0009fb0 tx=0x7f34a0009b30).stop 2024-01-13T11:50:27.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.776+0000 7f34b0f18700 1 -- 172.21.15.123:0/822719874 shutdown_connections 2024-01-13T11:50:27.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.776+0000 7f34b0f18700 1 --2- 172.21.15.123:0/822719874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34ac060280 0x7f34ac0f90f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:27.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.776+0000 7f34b0f18700 1 -- 172.21.15.123:0/822719874 >> 172.21.15.123:0/822719874 conn(0x7f34ac0f0b70 msgr2=0x7f34ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:27.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.776+0000 7f34b0f18700 1 -- 172.21.15.123:0/822719874 shutdown_connections 2024-01-13T11:50:27.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.777+0000 7f34b0f18700 1 -- 172.21.15.123:0/822719874 wait complete. 2024-01-13T11:50:27.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.777+0000 7f34b0f18700 1 Processor -- start 2024-01-13T11:50:27.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.777+0000 7f34b0f18700 1 -- start start 2024-01-13T11:50:27.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.777+0000 7f34b0f18700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34ac060280 0x7f34ac1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:27.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.777+0000 7f34b0f18700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac106bd0 con 0x7f34ac060280 2024-01-13T11:50:27.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.778+0000 7f34aa59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34ac060280 0x7f34ac1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:27.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.778+0000 7f34aa59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34ac060280 0x7f34ac1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:40068/0 (socket says 172.21.15.123:40068) 2024-01-13T11:50:27.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.778+0000 7f34aa59c700 1 -- 172.21.15.123:0/3537244767 learned_addr learned my addr 172.21.15.123:0/3537244767 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:50:27.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.778+0000 7f34aa59c700 1 -- 172.21.15.123:0/3537244767 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a0005040 con 0x7f34ac060280 2024-01-13T11:50:27.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.778+0000 7f34aa59c700 1 --2- 172.21.15.123:0/3537244767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34ac060280 0x7f34ac1066e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f34a0013510 tx=0x7f34a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:27.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.778+0000 7f349affd700 1 -- 172.21.15.123:0/3537244767 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34a0009320 con 0x7f34ac060280 2024-01-13T11:50:27.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.778+0000 7f34b0f18700 1 -- 172.21.15.123:0/3537244767 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34ac106dd0 con 0x7f34ac060280 2024-01-13T11:50:27.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.778+0000 7f349affd700 1 -- 172.21.15.123:0/3537244767 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34a0013f00 con 0x7f34ac060280 2024-01-13T11:50:27.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.778+0000 7f349affd700 1 -- 172.21.15.123:0/3537244767 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34a0014dc0 con 0x7f34ac060280 2024-01-13T11:50:27.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.778+0000 7f34b0f18700 1 -- 172.21.15.123:0/3537244767 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34ac107260 con 0x7f34ac060280 2024-01-13T11:50:27.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.779+0000 7f349affd700 1 -- 172.21.15.123:0/3537244767 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34a0014420 con 0x7f34ac060280 2024-01-13T11:50:27.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.779+0000 7f34b0f18700 1 -- 172.21.15.123:0/3537244767 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34ac045ad0 con 0x7f34ac060280 2024-01-13T11:50:27.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.779+0000 7f349affd700 1 --2- 172.21.15.123:0/3537244767 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f349402ff60 0x7f3494032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:27.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.779+0000 7f349affd700 1 -- 172.21.15.123:0/3537244767 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f34a0041050 con 0x7f34ac060280 2024-01-13T11:50:27.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.779+0000 7f34a9d9b700 1 --2- 172.21.15.123:0/3537244767 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f349402ff60 0x7f3494032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:27.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.780+0000 7f34a9d9b700 1 --2- 172.21.15.123:0/3537244767 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f349402ff60 0x7f3494032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f349c0096a0 tx=0x7f349c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:27.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.784+0000 7f349affd700 1 -- 172.21.15.123:0/3537244767 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34a00095f0 con 0x7f34ac060280 2024-01-13T11:50:27.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.991+0000 7f34b0f18700 1 -- 172.21.15.123:0/3537244767 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34ac108110 con 0x7f34ac060280 2024-01-13T11:50:27.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.992+0000 7f349affd700 1 -- 172.21.15.123:0/3537244767 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f34a001cb10 con 0x7f34ac060280 2024-01-13T11:50:27.981 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:50:27.981 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:50:27.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.994+0000 7f34b0f18700 1 -- 172.21.15.123:0/3537244767 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f349402ff60 msgr2=0x7f3494032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:27.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.994+0000 7f34b0f18700 1 --2- 172.21.15.123:0/3537244767 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f349402ff60 0x7f3494032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f349c0096a0 tx=0x7f349c006b40).stop 2024-01-13T11:50:27.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.994+0000 7f34b0f18700 1 -- 172.21.15.123:0/3537244767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34ac060280 msgr2=0x7f34ac1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:27.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.994+0000 7f34b0f18700 1 --2- 172.21.15.123:0/3537244767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34ac060280 0x7f34ac1066e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f34a0013510 tx=0x7f34a000b040).stop 2024-01-13T11:50:27.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.994+0000 7f34b0f18700 1 -- 172.21.15.123:0/3537244767 shutdown_connections 2024-01-13T11:50:27.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.994+0000 7f34b0f18700 1 --2- 172.21.15.123:0/3537244767 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f349402ff60 0x7f3494032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:27.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.994+0000 7f34b0f18700 1 --2- 172.21.15.123:0/3537244767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34ac060280 0x7f34ac1066e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:27.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.994+0000 7f34b0f18700 1 -- 172.21.15.123:0/3537244767 >> 172.21.15.123:0/3537244767 conn(0x7f34ac0f0b70 msgr2=0x7f34ac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:27.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.994+0000 7f34b0f18700 1 -- 172.21.15.123:0/3537244767 shutdown_connections 2024-01-13T11:50:27.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:27.994+0000 7f34b0f18700 1 -- 172.21.15.123:0/3537244767 wait complete. 2024-01-13T11:50:27.984 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:50:28.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:28 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:27.992332+0000 mon.smithi059 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.123:0/3537244767' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:50:29.358 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:50:29.358 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:50:29.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:29 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:27.507038+0000 mgr.smithi059.hmyldj (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:31.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:31 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:29.507256+0000 mgr.smithi059.hmyldj (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:31.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.841+0000 7fce738c4700 1 -- 172.21.15.123:0/2528301824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c0f5410 msgr2=0x7fce6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:31.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.841+0000 7fce738c4700 1 --2- 172.21.15.123:0/2528301824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c0f5410 0x7fce6c0f5830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fce5c009fb0 tx=0x7fce5c009b30).stop 2024-01-13T11:50:31.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.841+0000 7fce738c4700 1 -- 172.21.15.123:0/2528301824 shutdown_connections 2024-01-13T11:50:31.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.841+0000 7fce738c4700 1 --2- 172.21.15.123:0/2528301824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c0f5410 0x7fce6c0f5830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:31.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.841+0000 7fce738c4700 1 -- 172.21.15.123:0/2528301824 >> 172.21.15.123:0/2528301824 conn(0x7fce6c0f0b70 msgr2=0x7fce6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:31.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.841+0000 7fce738c4700 1 -- 172.21.15.123:0/2528301824 shutdown_connections 2024-01-13T11:50:31.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.841+0000 7fce738c4700 1 -- 172.21.15.123:0/2528301824 wait complete. 2024-01-13T11:50:31.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.842+0000 7fce738c4700 1 Processor -- start 2024-01-13T11:50:31.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.842+0000 7fce738c4700 1 -- start start 2024-01-13T11:50:31.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.842+0000 7fce738c4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c0f5410 0x7fce6c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:31.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.842+0000 7fce738c4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c105b20 con 0x7fce6c0f5410 2024-01-13T11:50:31.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.842+0000 7fce71660700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c0f5410 0x7fce6c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:31.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.842+0000 7fce71660700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c0f5410 0x7fce6c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:40094/0 (socket says 172.21.15.123:40094) 2024-01-13T11:50:31.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.842+0000 7fce71660700 1 -- 172.21.15.123:0/2282762731 learned_addr learned my addr 172.21.15.123:0/2282762731 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:50:31.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.842+0000 7fce71660700 1 -- 172.21.15.123:0/2282762731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5c005040 con 0x7fce6c0f5410 2024-01-13T11:50:31.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.842+0000 7fce71660700 1 --2- 172.21.15.123:0/2282762731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c0f5410 0x7fce6c105630 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fce5c013510 tx=0x7fce5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:31.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.842+0000 7fce61ffb700 1 -- 172.21.15.123:0/2282762731 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce5c009320 con 0x7fce6c0f5410 2024-01-13T11:50:31.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.843+0000 7fce738c4700 1 -- 172.21.15.123:0/2282762731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce6c105d20 con 0x7fce6c0f5410 2024-01-13T11:50:31.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.843+0000 7fce61ffb700 1 -- 172.21.15.123:0/2282762731 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce5c013f00 con 0x7fce6c0f5410 2024-01-13T11:50:31.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.843+0000 7fce61ffb700 1 -- 172.21.15.123:0/2282762731 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce5c014d90 con 0x7fce6c0f5410 2024-01-13T11:50:31.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.843+0000 7fce738c4700 1 -- 172.21.15.123:0/2282762731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce6c1061b0 con 0x7fce6c0f5410 2024-01-13T11:50:31.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.843+0000 7fce61ffb700 1 -- 172.21.15.123:0/2282762731 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce5c0095f0 con 0x7fce6c0f5410 2024-01-13T11:50:31.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.844+0000 7fce738c4700 1 -- 172.21.15.123:0/2282762731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce6c045ad0 con 0x7fce6c0f5410 2024-01-13T11:50:31.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.844+0000 7fce61ffb700 1 --2- 172.21.15.123:0/2282762731 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce5802ff60 0x7fce58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:31.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.844+0000 7fce61ffb700 1 -- 172.21.15.123:0/2282762731 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce5c041080 con 0x7fce6c0f5410 2024-01-13T11:50:31.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.844+0000 7fce70e5f700 1 --2- 172.21.15.123:0/2282762731 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce5802ff60 0x7fce58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:31.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.844+0000 7fce70e5f700 1 --2- 172.21.15.123:0/2282762731 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce5802ff60 0x7fce58032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fce680096a0 tx=0x7fce68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:31.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:31.849+0000 7fce61ffb700 1 -- 172.21.15.123:0/2282762731 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce5c014420 con 0x7fce6c0f5410 2024-01-13T11:50:32.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:32.054+0000 7fce738c4700 1 -- 172.21.15.123:0/2282762731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce6c0fae60 con 0x7fce6c0f5410 2024-01-13T11:50:32.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:32.054+0000 7fce61ffb700 1 -- 172.21.15.123:0/2282762731 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce5c010230 con 0x7fce6c0f5410 2024-01-13T11:50:32.044 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:50:32.044 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:50:32.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:32.056+0000 7fce738c4700 1 -- 172.21.15.123:0/2282762731 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce5802ff60 msgr2=0x7fce58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:32.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:32.056+0000 7fce738c4700 1 --2- 172.21.15.123:0/2282762731 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce5802ff60 0x7fce58032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fce680096a0 tx=0x7fce68006b40).stop 2024-01-13T11:50:32.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:32.056+0000 7fce738c4700 1 -- 172.21.15.123:0/2282762731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c0f5410 msgr2=0x7fce6c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:32.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:32.056+0000 7fce738c4700 1 --2- 172.21.15.123:0/2282762731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c0f5410 0x7fce6c105630 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fce5c013510 tx=0x7fce5c00b040).stop 2024-01-13T11:50:32.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:32.056+0000 7fce738c4700 1 -- 172.21.15.123:0/2282762731 shutdown_connections 2024-01-13T11:50:32.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:32.056+0000 7fce738c4700 1 --2- 172.21.15.123:0/2282762731 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce5802ff60 0x7fce58032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:32.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:32.056+0000 7fce738c4700 1 --2- 172.21.15.123:0/2282762731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c0f5410 0x7fce6c105630 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:32.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:32.056+0000 7fce738c4700 1 -- 172.21.15.123:0/2282762731 >> 172.21.15.123:0/2282762731 conn(0x7fce6c0f0b70 msgr2=0x7fce6c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:32.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:32.056+0000 7fce738c4700 1 -- 172.21.15.123:0/2282762731 shutdown_connections 2024-01-13T11:50:32.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:32.056+0000 7fce738c4700 1 -- 172.21.15.123:0/2282762731 wait complete. 2024-01-13T11:50:32.047 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:50:32.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:32 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:32.055611+0000 mon.smithi059 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.123:0/2282762731' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:50:33.435 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:50:33.435 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:50:33.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:33 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:31.507625+0000 mgr.smithi059.hmyldj (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:34.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.599+0000 7f7d74d13700 1 -- 172.21.15.123:0/3794923827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d700f5410 msgr2=0x7f7d700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:34.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.599+0000 7f7d74d13700 1 --2- 172.21.15.123:0/3794923827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d700f5410 0x7f7d700f5830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f7d64009fb0 tx=0x7f7d64009b30).stop 2024-01-13T11:50:34.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.599+0000 7f7d74d13700 1 -- 172.21.15.123:0/3794923827 shutdown_connections 2024-01-13T11:50:34.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.599+0000 7f7d74d13700 1 --2- 172.21.15.123:0/3794923827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d700f5410 0x7f7d700f5830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:34.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.599+0000 7f7d74d13700 1 -- 172.21.15.123:0/3794923827 >> 172.21.15.123:0/3794923827 conn(0x7f7d700f0b70 msgr2=0x7f7d700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:34.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.599+0000 7f7d74d13700 1 -- 172.21.15.123:0/3794923827 shutdown_connections 2024-01-13T11:50:34.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.599+0000 7f7d74d13700 1 -- 172.21.15.123:0/3794923827 wait complete. 2024-01-13T11:50:34.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.600+0000 7f7d74d13700 1 Processor -- start 2024-01-13T11:50:34.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.600+0000 7f7d74d13700 1 -- start start 2024-01-13T11:50:34.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.600+0000 7f7d74d13700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d700f5410 0x7f7d70105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:34.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.600+0000 7f7d74d13700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d70105b20 con 0x7f7d700f5410 2024-01-13T11:50:34.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.601+0000 7f7d6e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d700f5410 0x7f7d70105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:34.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.601+0000 7f7d6e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d700f5410 0x7f7d70105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:40110/0 (socket says 172.21.15.123:40110) 2024-01-13T11:50:34.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.601+0000 7f7d6e59c700 1 -- 172.21.15.123:0/3209914446 learned_addr learned my addr 172.21.15.123:0/3209914446 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:50:34.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.601+0000 7f7d6e59c700 1 -- 172.21.15.123:0/3209914446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d64005040 con 0x7f7d700f5410 2024-01-13T11:50:34.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.601+0000 7f7d6e59c700 1 --2- 172.21.15.123:0/3209914446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d700f5410 0x7f7d70105630 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f7d64013510 tx=0x7f7d6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:34.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.601+0000 7f7d5effd700 1 -- 172.21.15.123:0/3209914446 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d64009320 con 0x7f7d700f5410 2024-01-13T11:50:34.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.601+0000 7f7d74d13700 1 -- 172.21.15.123:0/3209914446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d70105d20 con 0x7f7d700f5410 2024-01-13T11:50:34.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.601+0000 7f7d5effd700 1 -- 172.21.15.123:0/3209914446 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d64013f00 con 0x7f7d700f5410 2024-01-13T11:50:34.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.601+0000 7f7d5effd700 1 -- 172.21.15.123:0/3209914446 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d64014d90 con 0x7f7d700f5410 2024-01-13T11:50:34.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.601+0000 7f7d74d13700 1 -- 172.21.15.123:0/3209914446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d701061b0 con 0x7f7d700f5410 2024-01-13T11:50:34.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.602+0000 7f7d5effd700 1 -- 172.21.15.123:0/3209914446 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7d640263a0 con 0x7f7d700f5410 2024-01-13T11:50:34.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.602+0000 7f7d5effd700 1 --2- 172.21.15.123:0/3209914446 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7d5802ff60 0x7f7d58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:34.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.602+0000 7f7d5effd700 1 -- 172.21.15.123:0/3209914446 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7d640410a0 con 0x7f7d700f5410 2024-01-13T11:50:34.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.602+0000 7f7d74d13700 1 -- 172.21.15.123:0/3209914446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d70045ad0 con 0x7f7d700f5410 2024-01-13T11:50:34.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.603+0000 7f7d6dd9b700 1 --2- 172.21.15.123:0/3209914446 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7d5802ff60 0x7f7d58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:34.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.603+0000 7f7d6dd9b700 1 --2- 172.21.15.123:0/3209914446 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7d5802ff60 0x7f7d58032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f7d600096a0 tx=0x7f7d60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:34.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.607+0000 7f7d5effd700 1 -- 172.21.15.123:0/3209914446 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d64012920 con 0x7f7d700f5410 2024-01-13T11:50:34.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.814+0000 7f7d74d13700 1 -- 172.21.15.123:0/3209914446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d700faf50 con 0x7f7d700f5410 2024-01-13T11:50:34.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.815+0000 7f7d5effd700 1 -- 172.21.15.123:0/3209914446 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7d6402b020 con 0x7f7d700f5410 2024-01-13T11:50:34.805 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:50:34.805 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:50:34.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.816+0000 7f7d74d13700 1 -- 172.21.15.123:0/3209914446 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7d5802ff60 msgr2=0x7f7d58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:34.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.816+0000 7f7d74d13700 1 --2- 172.21.15.123:0/3209914446 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7d5802ff60 0x7f7d58032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f7d600096a0 tx=0x7f7d60006b40).stop 2024-01-13T11:50:34.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.816+0000 7f7d74d13700 1 -- 172.21.15.123:0/3209914446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d700f5410 msgr2=0x7f7d70105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:34.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.816+0000 7f7d74d13700 1 --2- 172.21.15.123:0/3209914446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d700f5410 0x7f7d70105630 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f7d64013510 tx=0x7f7d6400b040).stop 2024-01-13T11:50:34.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.817+0000 7f7d74d13700 1 -- 172.21.15.123:0/3209914446 shutdown_connections 2024-01-13T11:50:34.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.817+0000 7f7d74d13700 1 --2- 172.21.15.123:0/3209914446 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7d5802ff60 0x7f7d58032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:34.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.817+0000 7f7d74d13700 1 --2- 172.21.15.123:0/3209914446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d700f5410 0x7f7d70105630 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:34.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.817+0000 7f7d74d13700 1 -- 172.21.15.123:0/3209914446 >> 172.21.15.123:0/3209914446 conn(0x7f7d700f0b70 msgr2=0x7f7d700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:34.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.817+0000 7f7d74d13700 1 -- 172.21.15.123:0/3209914446 shutdown_connections 2024-01-13T11:50:34.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:34.817+0000 7f7d74d13700 1 -- 172.21.15.123:0/3209914446 wait complete. 2024-01-13T11:50:34.807 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:50:35.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:35 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:33.507950+0000 mgr.smithi059.hmyldj (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:35.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:35 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:34.816178+0000 mon.smithi059 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.123:0/3209914446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:50:36.179 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:50:36.180 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:50:37.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.287+0000 7ff960015700 1 -- 172.21.15.123:0/2969910767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9580601e0 msgr2=0x7ff9580f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:37.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.287+0000 7ff960015700 1 --2- 172.21.15.123:0/2969910767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9580601e0 0x7ff9580f9190 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7ff948009fb0 tx=0x7ff948009b30).stop 2024-01-13T11:50:37.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.287+0000 7ff960015700 1 -- 172.21.15.123:0/2969910767 shutdown_connections 2024-01-13T11:50:37.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.287+0000 7ff960015700 1 --2- 172.21.15.123:0/2969910767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9580601e0 0x7ff9580f9190 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:37.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.287+0000 7ff960015700 1 -- 172.21.15.123:0/2969910767 >> 172.21.15.123:0/2969910767 conn(0x7ff9580f0b70 msgr2=0x7ff9580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:37.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.287+0000 7ff960015700 1 -- 172.21.15.123:0/2969910767 shutdown_connections 2024-01-13T11:50:37.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.287+0000 7ff960015700 1 -- 172.21.15.123:0/2969910767 wait complete. 2024-01-13T11:50:37.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.288+0000 7ff960015700 1 Processor -- start 2024-01-13T11:50:37.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.288+0000 7ff960015700 1 -- start start 2024-01-13T11:50:37.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.288+0000 7ff960015700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9580601e0 0x7ff9581066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:37.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.288+0000 7ff960015700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff958106bd0 con 0x7ff9580601e0 2024-01-13T11:50:37.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.289+0000 7ff95ddb1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9580601e0 0x7ff9581066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:37.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.289+0000 7ff95ddb1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9580601e0 0x7ff9581066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:44630/0 (socket says 172.21.15.123:44630) 2024-01-13T11:50:37.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.289+0000 7ff95ddb1700 1 -- 172.21.15.123:0/2655374839 learned_addr learned my addr 172.21.15.123:0/2655374839 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:50:37.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.289+0000 7ff95ddb1700 1 -- 172.21.15.123:0/2655374839 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff948005040 con 0x7ff9580601e0 2024-01-13T11:50:37.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.289+0000 7ff95ddb1700 1 --2- 172.21.15.123:0/2655374839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9580601e0 0x7ff9581066e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7ff9480079e0 tx=0x7ff948009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:37.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.289+0000 7ff94e7fc700 1 -- 172.21.15.123:0/2655374839 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff948018070 con 0x7ff9580601e0 2024-01-13T11:50:37.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.289+0000 7ff94e7fc700 1 -- 172.21.15.123:0/2655374839 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff948013410 con 0x7ff9580601e0 2024-01-13T11:50:37.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.289+0000 7ff960015700 1 -- 172.21.15.123:0/2655374839 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff958106dd0 con 0x7ff9580601e0 2024-01-13T11:50:37.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.289+0000 7ff94e7fc700 1 -- 172.21.15.123:0/2655374839 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff94801c3c0 con 0x7ff9580601e0 2024-01-13T11:50:37.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.289+0000 7ff960015700 1 -- 172.21.15.123:0/2655374839 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff958107260 con 0x7ff9580601e0 2024-01-13T11:50:37.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.290+0000 7ff94e7fc700 1 -- 172.21.15.123:0/2655374839 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff94801d420 con 0x7ff9580601e0 2024-01-13T11:50:37.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.290+0000 7ff960015700 1 -- 172.21.15.123:0/2655374839 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff958045ad0 con 0x7ff9580601e0 2024-01-13T11:50:37.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.290+0000 7ff94e7fc700 1 --2- 172.21.15.123:0/2655374839 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff94402ff60 0x7ff944032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:37.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.290+0000 7ff94e7fc700 1 -- 172.21.15.123:0/2655374839 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff9480413a0 con 0x7ff9580601e0 2024-01-13T11:50:37.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.290+0000 7ff95d5b0700 1 --2- 172.21.15.123:0/2655374839 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff94402ff60 0x7ff944032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:37.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.291+0000 7ff95d5b0700 1 --2- 172.21.15.123:0/2655374839 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff94402ff60 0x7ff944032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7ff9540096a0 tx=0x7ff954006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:37.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.295+0000 7ff94e7fc700 1 -- 172.21.15.123:0/2655374839 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff94801d7e0 con 0x7ff9580601e0 2024-01-13T11:50:37.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.506+0000 7ff960015700 1 -- 172.21.15.123:0/2655374839 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9581080a0 con 0x7ff9580601e0 2024-01-13T11:50:37.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.507+0000 7ff94e7fc700 1 -- 172.21.15.123:0/2655374839 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff94801b370 con 0x7ff9580601e0 2024-01-13T11:50:37.497 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:50:37.497 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:50:37.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.509+0000 7ff960015700 1 -- 172.21.15.123:0/2655374839 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff94402ff60 msgr2=0x7ff944032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:37.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.509+0000 7ff960015700 1 --2- 172.21.15.123:0/2655374839 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff94402ff60 0x7ff944032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7ff9540096a0 tx=0x7ff954006b40).stop 2024-01-13T11:50:37.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.509+0000 7ff960015700 1 -- 172.21.15.123:0/2655374839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9580601e0 msgr2=0x7ff9581066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:37.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.509+0000 7ff960015700 1 --2- 172.21.15.123:0/2655374839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9580601e0 0x7ff9581066e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7ff9480079e0 tx=0x7ff948009b30).stop 2024-01-13T11:50:37.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.509+0000 7ff960015700 1 -- 172.21.15.123:0/2655374839 shutdown_connections 2024-01-13T11:50:37.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.509+0000 7ff960015700 1 --2- 172.21.15.123:0/2655374839 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff94402ff60 0x7ff944032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:37.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.509+0000 7ff960015700 1 --2- 172.21.15.123:0/2655374839 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9580601e0 0x7ff9581066e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:37.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.509+0000 7ff960015700 1 -- 172.21.15.123:0/2655374839 >> 172.21.15.123:0/2655374839 conn(0x7ff9580f0b70 msgr2=0x7ff9580f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:37.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.509+0000 7ff960015700 1 -- 172.21.15.123:0/2655374839 shutdown_connections 2024-01-13T11:50:37.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:37.509+0000 7ff960015700 1 -- 172.21.15.123:0/2655374839 wait complete. 2024-01-13T11:50:37.499 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:50:37.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:37 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:35.508254+0000 mgr.smithi059.hmyldj (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:38.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:38 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:37.508324+0000 mon.smithi059 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.123:0/2655374839' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:50:38.915 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:50:38.916 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:50:39.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:39 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:37.508532+0000 mgr.smithi059.hmyldj (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:40.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.060+0000 7f052b4c6700 1 -- 172.21.15.123:0/397289705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05240f6510 msgr2=0x7f05240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:40.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.060+0000 7f052b4c6700 1 --2- 172.21.15.123:0/397289705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05240f6510 0x7f05240f6930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f0514009fb0 tx=0x7f0514009b30).stop 2024-01-13T11:50:40.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.060+0000 7f052b4c6700 1 -- 172.21.15.123:0/397289705 shutdown_connections 2024-01-13T11:50:40.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.060+0000 7f052b4c6700 1 --2- 172.21.15.123:0/397289705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05240f6510 0x7f05240f6930 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:40.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.060+0000 7f052b4c6700 1 -- 172.21.15.123:0/397289705 >> 172.21.15.123:0/397289705 conn(0x7f05240f1c70 msgr2=0x7f05240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:40.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.060+0000 7f052b4c6700 1 -- 172.21.15.123:0/397289705 shutdown_connections 2024-01-13T11:50:40.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.060+0000 7f052b4c6700 1 -- 172.21.15.123:0/397289705 wait complete. 2024-01-13T11:50:40.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.061+0000 7f052b4c6700 1 Processor -- start 2024-01-13T11:50:40.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.061+0000 7f052b4c6700 1 -- start start 2024-01-13T11:50:40.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.061+0000 7f052b4c6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05240f6510 0x7f0524106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:40.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.061+0000 7f052b4c6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0524106c20 con 0x7f05240f6510 2024-01-13T11:50:40.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.061+0000 7f0529262700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05240f6510 0x7f0524106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:40.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.061+0000 7f0529262700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05240f6510 0x7f0524106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:44644/0 (socket says 172.21.15.123:44644) 2024-01-13T11:50:40.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.061+0000 7f0529262700 1 -- 172.21.15.123:0/112481835 learned_addr learned my addr 172.21.15.123:0/112481835 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:50:40.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.061+0000 7f0529262700 1 -- 172.21.15.123:0/112481835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0514005040 con 0x7f05240f6510 2024-01-13T11:50:40.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.061+0000 7f0529262700 1 --2- 172.21.15.123:0/112481835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05240f6510 0x7f0524106730 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f0514013510 tx=0x7f051400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:40.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.062+0000 7f0519ffb700 1 -- 172.21.15.123:0/112481835 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0514009320 con 0x7f05240f6510 2024-01-13T11:50:40.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.062+0000 7f052b4c6700 1 -- 172.21.15.123:0/112481835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0524106e20 con 0x7f05240f6510 2024-01-13T11:50:40.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.062+0000 7f0519ffb700 1 -- 172.21.15.123:0/112481835 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0514013f00 con 0x7f05240f6510 2024-01-13T11:50:40.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.062+0000 7f0519ffb700 1 -- 172.21.15.123:0/112481835 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0514014d90 con 0x7f05240f6510 2024-01-13T11:50:40.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.062+0000 7f052b4c6700 1 -- 172.21.15.123:0/112481835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05241072b0 con 0x7f05240f6510 2024-01-13T11:50:40.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.063+0000 7f0519ffb700 1 -- 172.21.15.123:0/112481835 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f05140095f0 con 0x7f05240f6510 2024-01-13T11:50:40.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.063+0000 7f052b4c6700 1 -- 172.21.15.123:0/112481835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0524045ad0 con 0x7f05240f6510 2024-01-13T11:50:40.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.063+0000 7f0519ffb700 1 --2- 172.21.15.123:0/112481835 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f051002ffb0 0x7f0510032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:40.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.063+0000 7f0519ffb700 1 -- 172.21.15.123:0/112481835 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0514040f80 con 0x7f05240f6510 2024-01-13T11:50:40.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.063+0000 7f0528a61700 1 --2- 172.21.15.123:0/112481835 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f051002ffb0 0x7f0510032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:40.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.063+0000 7f0528a61700 1 --2- 172.21.15.123:0/112481835 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f051002ffb0 0x7f0510032470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f05200096a0 tx=0x7f0520006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:40.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.067+0000 7f0519ffb700 1 -- 172.21.15.123:0/112481835 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0514014420 con 0x7f05240f6510 2024-01-13T11:50:40.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.276+0000 7f052b4c6700 1 -- 172.21.15.123:0/112481835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05240fbf60 con 0x7f05240f6510 2024-01-13T11:50:40.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.277+0000 7f0519ffb700 1 -- 172.21.15.123:0/112481835 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f051401d7f0 con 0x7f05240f6510 2024-01-13T11:50:40.266 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:50:40.266 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:50:40.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.279+0000 7f052b4c6700 1 -- 172.21.15.123:0/112481835 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f051002ffb0 msgr2=0x7f0510032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:40.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.279+0000 7f052b4c6700 1 --2- 172.21.15.123:0/112481835 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f051002ffb0 0x7f0510032470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f05200096a0 tx=0x7f0520006b40).stop 2024-01-13T11:50:40.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.279+0000 7f052b4c6700 1 -- 172.21.15.123:0/112481835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05240f6510 msgr2=0x7f0524106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:40.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.279+0000 7f052b4c6700 1 --2- 172.21.15.123:0/112481835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05240f6510 0x7f0524106730 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f0514013510 tx=0x7f051400b040).stop 2024-01-13T11:50:40.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.279+0000 7f052b4c6700 1 -- 172.21.15.123:0/112481835 shutdown_connections 2024-01-13T11:50:40.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.279+0000 7f052b4c6700 1 --2- 172.21.15.123:0/112481835 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f051002ffb0 0x7f0510032470 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:40.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.279+0000 7f052b4c6700 1 --2- 172.21.15.123:0/112481835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05240f6510 0x7f0524106730 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:40.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.279+0000 7f052b4c6700 1 -- 172.21.15.123:0/112481835 >> 172.21.15.123:0/112481835 conn(0x7f05240f1c70 msgr2=0x7f05240f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:40.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.279+0000 7f052b4c6700 1 -- 172.21.15.123:0/112481835 shutdown_connections 2024-01-13T11:50:40.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:40.279+0000 7f052b4c6700 1 -- 172.21.15.123:0/112481835 wait complete. 2024-01-13T11:50:40.269 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:50:40.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:39.637244+0000 mon.smithi059 (mon.0) 127 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/mirror_snapshot_schedule"}]: dispatch 2024-01-13T11:50:40.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:39.639624+0000 mon.smithi059 (mon.0) 128 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/trash_purge_schedule"}]: dispatch 2024-01-13T11:50:41.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:41 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:39.508774+0000 mgr.smithi059.hmyldj (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:41.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:41 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:40.277874+0000 mon.smithi059 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.123:0/112481835' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:50:41.645 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:50:41.645 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:50:43.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:43 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:41.509139+0000 mgr.smithi059.hmyldj (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:45.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:45 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:43.509538+0000 mgr.smithi059.hmyldj (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:46.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.496+0000 7f91c359e700 1 -- 172.21.15.123:0/2369314969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91c40f62a0 msgr2=0x7f91c40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:46.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.496+0000 7f91c359e700 1 --2- 172.21.15.123:0/2369314969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91c40f62a0 0x7f91c40f66c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f91b8009fb0 tx=0x7f91b8009b30).stop 2024-01-13T11:50:46.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.496+0000 7f91c359e700 1 -- 172.21.15.123:0/2369314969 shutdown_connections 2024-01-13T11:50:46.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.496+0000 7f91c359e700 1 --2- 172.21.15.123:0/2369314969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91c40f62a0 0x7f91c40f66c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:46.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.496+0000 7f91c359e700 1 -- 172.21.15.123:0/2369314969 >> 172.21.15.123:0/2369314969 conn(0x7f91c40f1aa0 msgr2=0x7f91c40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:46.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.496+0000 7f91c359e700 1 -- 172.21.15.123:0/2369314969 shutdown_connections 2024-01-13T11:50:46.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.496+0000 7f91c359e700 1 -- 172.21.15.123:0/2369314969 wait complete. 2024-01-13T11:50:46.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.496+0000 7f91c359e700 1 Processor -- start 2024-01-13T11:50:46.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.497+0000 7f91c359e700 1 -- start start 2024-01-13T11:50:46.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.497+0000 7f91c359e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91c40f62a0 0x7f91c41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:46.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.497+0000 7f91c359e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c41069a0 con 0x7f91c40f62a0 2024-01-13T11:50:46.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.497+0000 7f91c259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91c40f62a0 0x7f91c41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:46.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.497+0000 7f91c259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91c40f62a0 0x7f91c41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:47728/0 (socket says 172.21.15.123:47728) 2024-01-13T11:50:46.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.497+0000 7f91c259c700 1 -- 172.21.15.123:0/2193760899 learned_addr learned my addr 172.21.15.123:0/2193760899 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:50:46.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.497+0000 7f91c259c700 1 -- 172.21.15.123:0/2193760899 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b8005040 con 0x7f91c40f62a0 2024-01-13T11:50:46.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.497+0000 7f91c259c700 1 --2- 172.21.15.123:0/2193760899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91c40f62a0 0x7f91c41064b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f91b8013510 tx=0x7f91b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:46.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.497+0000 7f91b2ffd700 1 -- 172.21.15.123:0/2193760899 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91b8009320 con 0x7f91c40f62a0 2024-01-13T11:50:46.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.498+0000 7f91b2ffd700 1 -- 172.21.15.123:0/2193760899 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91b8013f00 con 0x7f91c40f62a0 2024-01-13T11:50:46.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.498+0000 7f91c359e700 1 -- 172.21.15.123:0/2193760899 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c4106ba0 con 0x7f91c40f62a0 2024-01-13T11:50:46.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.498+0000 7f91b2ffd700 1 -- 172.21.15.123:0/2193760899 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91b8014dc0 con 0x7f91c40f62a0 2024-01-13T11:50:46.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.498+0000 7f91c359e700 1 -- 172.21.15.123:0/2193760899 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c4107030 con 0x7f91c40f62a0 2024-01-13T11:50:46.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.498+0000 7f91b2ffd700 1 -- 172.21.15.123:0/2193760899 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f91b80095f0 con 0x7f91c40f62a0 2024-01-13T11:50:46.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.498+0000 7f91c359e700 1 -- 172.21.15.123:0/2193760899 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c4045ad0 con 0x7f91c40f62a0 2024-01-13T11:50:46.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.499+0000 7f91b2ffd700 1 --2- 172.21.15.123:0/2193760899 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f91ac02ff60 0x7f91ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:46.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.499+0000 7f91b2ffd700 1 -- 172.21.15.123:0/2193760899 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f91b8041080 con 0x7f91c40f62a0 2024-01-13T11:50:46.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.499+0000 7f91c1d9b700 1 --2- 172.21.15.123:0/2193760899 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f91ac02ff60 0x7f91ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:46.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.499+0000 7f91c1d9b700 1 --2- 172.21.15.123:0/2193760899 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f91ac02ff60 0x7f91ac032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f91b40096a0 tx=0x7f91b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:46.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.504+0000 7f91b2ffd700 1 -- 172.21.15.123:0/2193760899 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f91b801c790 con 0x7f91c40f62a0 2024-01-13T11:50:46.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.710+0000 7f91c359e700 1 -- 172.21.15.123:0/2193760899 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f91c4107ee0 con 0x7f91c40f62a0 2024-01-13T11:50:46.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.710+0000 7f91b2ffd700 1 -- 172.21.15.123:0/2193760899 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f91b8010230 con 0x7f91c40f62a0 2024-01-13T11:50:46.700 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:50:46.700 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:50:46.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.712+0000 7f91c359e700 1 -- 172.21.15.123:0/2193760899 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f91ac02ff60 msgr2=0x7f91ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:46.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.712+0000 7f91c359e700 1 --2- 172.21.15.123:0/2193760899 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f91ac02ff60 0x7f91ac032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f91b40096a0 tx=0x7f91b4006b40).stop 2024-01-13T11:50:46.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.712+0000 7f91c359e700 1 -- 172.21.15.123:0/2193760899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91c40f62a0 msgr2=0x7f91c41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:46.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.712+0000 7f91c359e700 1 --2- 172.21.15.123:0/2193760899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91c40f62a0 0x7f91c41064b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f91b8013510 tx=0x7f91b800b040).stop 2024-01-13T11:50:46.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.712+0000 7f91c359e700 1 -- 172.21.15.123:0/2193760899 shutdown_connections 2024-01-13T11:50:46.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.712+0000 7f91c359e700 1 --2- 172.21.15.123:0/2193760899 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f91ac02ff60 0x7f91ac032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:46.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.712+0000 7f91c359e700 1 --2- 172.21.15.123:0/2193760899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91c40f62a0 0x7f91c41064b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:46.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.712+0000 7f91c359e700 1 -- 172.21.15.123:0/2193760899 >> 172.21.15.123:0/2193760899 conn(0x7f91c40f1aa0 msgr2=0x7f91c40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:46.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.712+0000 7f91c359e700 1 -- 172.21.15.123:0/2193760899 shutdown_connections 2024-01-13T11:50:46.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:46.712+0000 7f91c359e700 1 -- 172.21.15.123:0/2193760899 wait complete. 2024-01-13T11:50:46.703 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:50:47.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:47 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:45.509892+0000 mgr.smithi059.hmyldj (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:47.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:47 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:46.711581+0000 mon.smithi059 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.123:0/2193760899' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:50:49.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:49 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:47.510240+0000 mgr.smithi059.hmyldj (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:50.261 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:50:50.262 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:50:51.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.437+0000 7f8d53937700 1 -- 172.21.15.123:0/3723444952 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d4c0f5410 msgr2=0x7f8d4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:51.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.437+0000 7f8d53937700 1 --2- 172.21.15.123:0/3723444952 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d4c0f5410 0x7f8d4c0f5830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f8d48009fb0 tx=0x7f8d48009b30).stop 2024-01-13T11:50:51.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.437+0000 7f8d53937700 1 -- 172.21.15.123:0/3723444952 shutdown_connections 2024-01-13T11:50:51.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.437+0000 7f8d53937700 1 --2- 172.21.15.123:0/3723444952 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d4c0f5410 0x7f8d4c0f5830 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:51.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.437+0000 7f8d53937700 1 -- 172.21.15.123:0/3723444952 >> 172.21.15.123:0/3723444952 conn(0x7f8d4c0f0b70 msgr2=0x7f8d4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:51.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.437+0000 7f8d53937700 1 -- 172.21.15.123:0/3723444952 shutdown_connections 2024-01-13T11:50:51.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.437+0000 7f8d53937700 1 -- 172.21.15.123:0/3723444952 wait complete. 2024-01-13T11:50:51.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.438+0000 7f8d53937700 1 Processor -- start 2024-01-13T11:50:51.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.438+0000 7f8d53937700 1 -- start start 2024-01-13T11:50:51.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.438+0000 7f8d53937700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d4c0f5410 0x7f8d4c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.438+0000 7f8d53937700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c060240 con 0x7f8d4c0f5410 2024-01-13T11:50:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.438+0000 7f8d516d3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d4c0f5410 0x7f8d4c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.438+0000 7f8d516d3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d4c0f5410 0x7f8d4c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:47734/0 (socket says 172.21.15.123:47734) 2024-01-13T11:50:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.438+0000 7f8d516d3700 1 -- 172.21.15.123:0/3310871908 learned_addr learned my addr 172.21.15.123:0/3310871908 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:50:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.438+0000 7f8d516d3700 1 -- 172.21.15.123:0/3310871908 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d48005040 con 0x7f8d4c0f5410 2024-01-13T11:50:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.438+0000 7f8d516d3700 1 --2- 172.21.15.123:0/3310871908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d4c0f5410 0x7f8d4c109430 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f8d48013510 tx=0x7f8d4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.439+0000 7f8d41ffb700 1 -- 172.21.15.123:0/3310871908 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d48009320 con 0x7f8d4c0f5410 2024-01-13T11:50:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.439+0000 7f8d53937700 1 -- 172.21.15.123:0/3310871908 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d4c060440 con 0x7f8d4c0f5410 2024-01-13T11:50:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.439+0000 7f8d41ffb700 1 -- 172.21.15.123:0/3310871908 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d48013f00 con 0x7f8d4c0f5410 2024-01-13T11:50:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.439+0000 7f8d41ffb700 1 -- 172.21.15.123:0/3310871908 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d48014d90 con 0x7f8d4c0f5410 2024-01-13T11:50:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.439+0000 7f8d53937700 1 -- 172.21.15.123:0/3310871908 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d4c109ae0 con 0x7f8d4c0f5410 2024-01-13T11:50:51.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.440+0000 7f8d41ffb700 1 -- 172.21.15.123:0/3310871908 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8d480095f0 con 0x7f8d4c0f5410 2024-01-13T11:50:51.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.440+0000 7f8d53937700 1 -- 172.21.15.123:0/3310871908 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4c045ad0 con 0x7f8d4c0f5410 2024-01-13T11:50:51.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.440+0000 7f8d41ffb700 1 --2- 172.21.15.123:0/3310871908 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8d3802ff60 0x7f8d38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:51.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.440+0000 7f8d41ffb700 1 -- 172.21.15.123:0/3310871908 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8d480410c0 con 0x7f8d4c0f5410 2024-01-13T11:50:51.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.440+0000 7f8d50ed2700 1 --2- 172.21.15.123:0/3310871908 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8d3802ff60 0x7f8d38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:51.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.440+0000 7f8d50ed2700 1 --2- 172.21.15.123:0/3310871908 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8d3802ff60 0x7f8d38032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f8d3c0096a0 tx=0x7f8d3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:51.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.445+0000 7f8d41ffb700 1 -- 172.21.15.123:0/3310871908 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d4801d7f0 con 0x7f8d4c0f5410 2024-01-13T11:50:51.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:49.510627+0000 mgr.smithi059.hmyldj (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:51.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.651+0000 7f8d53937700 1 -- 172.21.15.123:0/3310871908 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d4c0fae60 con 0x7f8d4c0f5410 2024-01-13T11:50:51.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.651+0000 7f8d41ffb700 1 -- 172.21.15.123:0/3310871908 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8d48010230 con 0x7f8d4c0f5410 2024-01-13T11:50:51.641 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:50:51.641 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:50:51.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.653+0000 7f8d53937700 1 -- 172.21.15.123:0/3310871908 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8d3802ff60 msgr2=0x7f8d38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:51.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.653+0000 7f8d53937700 1 --2- 172.21.15.123:0/3310871908 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8d3802ff60 0x7f8d38032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f8d3c0096a0 tx=0x7f8d3c006b40).stop 2024-01-13T11:50:51.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.653+0000 7f8d53937700 1 -- 172.21.15.123:0/3310871908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d4c0f5410 msgr2=0x7f8d4c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:51.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.653+0000 7f8d53937700 1 --2- 172.21.15.123:0/3310871908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d4c0f5410 0x7f8d4c109430 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f8d48013510 tx=0x7f8d4800b040).stop 2024-01-13T11:50:51.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.653+0000 7f8d53937700 1 -- 172.21.15.123:0/3310871908 shutdown_connections 2024-01-13T11:50:51.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.653+0000 7f8d53937700 1 --2- 172.21.15.123:0/3310871908 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8d3802ff60 0x7f8d38032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:51.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.653+0000 7f8d53937700 1 --2- 172.21.15.123:0/3310871908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d4c0f5410 0x7f8d4c109430 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:51.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.653+0000 7f8d53937700 1 -- 172.21.15.123:0/3310871908 >> 172.21.15.123:0/3310871908 conn(0x7f8d4c0f0b70 msgr2=0x7f8d4c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:51.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.653+0000 7f8d53937700 1 -- 172.21.15.123:0/3310871908 shutdown_connections 2024-01-13T11:50:51.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:51.653+0000 7f8d53937700 1 -- 172.21.15.123:0/3310871908 wait complete. 2024-01-13T11:50:51.644 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:50:52.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:51.652737+0000 mon.smithi059 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.123:0/3310871908' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:50:53.006 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:50:53.007 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:50:53.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:53 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:51.510829+0000 mgr.smithi059.hmyldj (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:54.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.179+0000 7f632c890700 1 -- 172.21.15.123:0/2420035027 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63280f5410 msgr2=0x7f63280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:54.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.179+0000 7f632c890700 1 --2- 172.21.15.123:0/2420035027 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63280f5410 0x7f63280f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f6310009fb0 tx=0x7f6310009b30).stop 2024-01-13T11:50:54.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.179+0000 7f632c890700 1 -- 172.21.15.123:0/2420035027 shutdown_connections 2024-01-13T11:50:54.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.179+0000 7f632c890700 1 --2- 172.21.15.123:0/2420035027 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63280f5410 0x7f63280f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:54.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.179+0000 7f632c890700 1 -- 172.21.15.123:0/2420035027 >> 172.21.15.123:0/2420035027 conn(0x7f63280f0b70 msgr2=0x7f63280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:54.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.179+0000 7f632c890700 1 -- 172.21.15.123:0/2420035027 shutdown_connections 2024-01-13T11:50:54.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.179+0000 7f632c890700 1 -- 172.21.15.123:0/2420035027 wait complete. 2024-01-13T11:50:54.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.180+0000 7f632c890700 1 Processor -- start 2024-01-13T11:50:54.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.180+0000 7f632c890700 1 -- start start 2024-01-13T11:50:54.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.180+0000 7f632c890700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63280f5410 0x7f6328105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:54.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.180+0000 7f632c890700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6328105b20 con 0x7f63280f5410 2024-01-13T11:50:54.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.180+0000 7f632659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63280f5410 0x7f6328105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:54.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.180+0000 7f632659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63280f5410 0x7f6328105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:47752/0 (socket says 172.21.15.123:47752) 2024-01-13T11:50:54.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.180+0000 7f632659c700 1 -- 172.21.15.123:0/680179000 learned_addr learned my addr 172.21.15.123:0/680179000 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:50:54.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.181+0000 7f632659c700 1 -- 172.21.15.123:0/680179000 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6310005040 con 0x7f63280f5410 2024-01-13T11:50:54.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.181+0000 7f632659c700 1 --2- 172.21.15.123:0/680179000 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63280f5410 0x7f6328105630 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f6310013510 tx=0x7f631000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:54.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.181+0000 7f631effd700 1 -- 172.21.15.123:0/680179000 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6310009320 con 0x7f63280f5410 2024-01-13T11:50:54.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.181+0000 7f632c890700 1 -- 172.21.15.123:0/680179000 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6328105d20 con 0x7f63280f5410 2024-01-13T11:50:54.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.181+0000 7f631effd700 1 -- 172.21.15.123:0/680179000 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6310013f00 con 0x7f63280f5410 2024-01-13T11:50:54.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.181+0000 7f631effd700 1 -- 172.21.15.123:0/680179000 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6310014d90 con 0x7f63280f5410 2024-01-13T11:50:54.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.181+0000 7f632c890700 1 -- 172.21.15.123:0/680179000 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63281061b0 con 0x7f63280f5410 2024-01-13T11:50:54.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.182+0000 7f631effd700 1 -- 172.21.15.123:0/680179000 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6310014420 con 0x7f63280f5410 2024-01-13T11:50:54.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.182+0000 7f632c890700 1 -- 172.21.15.123:0/680179000 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6328045ad0 con 0x7f63280f5410 2024-01-13T11:50:54.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.182+0000 7f631effd700 1 --2- 172.21.15.123:0/680179000 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f631402ff60 0x7f6314032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:54.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.182+0000 7f631effd700 1 -- 172.21.15.123:0/680179000 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6310041000 con 0x7f63280f5410 2024-01-13T11:50:54.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.182+0000 7f6325d9b700 1 --2- 172.21.15.123:0/680179000 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f631402ff60 0x7f6314032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:54.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.183+0000 7f6325d9b700 1 --2- 172.21.15.123:0/680179000 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f631402ff60 0x7f6314032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f63180096a0 tx=0x7f6318006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:54.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.187+0000 7f631effd700 1 -- 172.21.15.123:0/680179000 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63100095f0 con 0x7f63280f5410 2024-01-13T11:50:54.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.394+0000 7f632c890700 1 -- 172.21.15.123:0/680179000 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63280fae60 con 0x7f63280f5410 2024-01-13T11:50:54.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.395+0000 7f631effd700 1 -- 172.21.15.123:0/680179000 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f631001db70 con 0x7f63280f5410 2024-01-13T11:50:54.385 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:50:54.385 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:50:54.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.396+0000 7f632c890700 1 -- 172.21.15.123:0/680179000 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f631402ff60 msgr2=0x7f6314032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:54.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.396+0000 7f632c890700 1 --2- 172.21.15.123:0/680179000 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f631402ff60 0x7f6314032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f63180096a0 tx=0x7f6318006b40).stop 2024-01-13T11:50:54.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.396+0000 7f632c890700 1 -- 172.21.15.123:0/680179000 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63280f5410 msgr2=0x7f6328105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:54.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.396+0000 7f632c890700 1 --2- 172.21.15.123:0/680179000 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63280f5410 0x7f6328105630 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f6310013510 tx=0x7f631000b040).stop 2024-01-13T11:50:54.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.396+0000 7f632c890700 1 -- 172.21.15.123:0/680179000 shutdown_connections 2024-01-13T11:50:54.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.396+0000 7f632c890700 1 --2- 172.21.15.123:0/680179000 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f631402ff60 0x7f6314032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:54.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.396+0000 7f632c890700 1 --2- 172.21.15.123:0/680179000 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63280f5410 0x7f6328105630 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:54.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.396+0000 7f632c890700 1 -- 172.21.15.123:0/680179000 >> 172.21.15.123:0/680179000 conn(0x7f63280f0b70 msgr2=0x7f63280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:54.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.396+0000 7f632c890700 1 -- 172.21.15.123:0/680179000 shutdown_connections 2024-01-13T11:50:54.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:54.396+0000 7f632c890700 1 -- 172.21.15.123:0/680179000 wait complete. 2024-01-13T11:50:54.387 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:50:55.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:55 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:53.511160+0000 mgr.smithi059.hmyldj (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:55.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:55 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:54.396084+0000 mon.smithi059 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.123:0/680179000' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:50:55.759 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:50:55.760 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:50:56.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.887+0000 7f6d7f050700 1 -- 172.21.15.123:0/3763202087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d780f5410 msgr2=0x7f6d780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:56.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.887+0000 7f6d7f050700 1 --2- 172.21.15.123:0/3763202087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d780f5410 0x7f6d780f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f6d6c009fb0 tx=0x7f6d6c009b30).stop 2024-01-13T11:50:56.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.887+0000 7f6d7f050700 1 -- 172.21.15.123:0/3763202087 shutdown_connections 2024-01-13T11:50:56.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.887+0000 7f6d7f050700 1 --2- 172.21.15.123:0/3763202087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d780f5410 0x7f6d780f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:56.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.887+0000 7f6d7f050700 1 -- 172.21.15.123:0/3763202087 >> 172.21.15.123:0/3763202087 conn(0x7f6d780f0b70 msgr2=0x7f6d780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:56.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.887+0000 7f6d7f050700 1 -- 172.21.15.123:0/3763202087 shutdown_connections 2024-01-13T11:50:56.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.887+0000 7f6d7f050700 1 -- 172.21.15.123:0/3763202087 wait complete. 2024-01-13T11:50:56.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.888+0000 7f6d7f050700 1 Processor -- start 2024-01-13T11:50:56.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.888+0000 7f6d7f050700 1 -- start start 2024-01-13T11:50:56.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.888+0000 7f6d7f050700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d780f5410 0x7f6d78105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:56.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.888+0000 7f6d7f050700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d78105b20 con 0x7f6d780f5410 2024-01-13T11:50:56.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.888+0000 7f6d7cdec700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d780f5410 0x7f6d78105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:56.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.888+0000 7f6d7cdec700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d780f5410 0x7f6d78105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:41410/0 (socket says 172.21.15.123:41410) 2024-01-13T11:50:56.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.888+0000 7f6d7cdec700 1 -- 172.21.15.123:0/356139753 learned_addr learned my addr 172.21.15.123:0/356139753 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:50:56.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.889+0000 7f6d7cdec700 1 -- 172.21.15.123:0/356139753 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6c005040 con 0x7f6d780f5410 2024-01-13T11:50:56.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.889+0000 7f6d7cdec700 1 --2- 172.21.15.123:0/356139753 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d780f5410 0x7f6d78105630 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f6d6c014000 tx=0x7f6d6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:56.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.889+0000 7f6d757fa700 1 -- 172.21.15.123:0/356139753 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d6c0095f0 con 0x7f6d780f5410 2024-01-13T11:50:56.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.889+0000 7f6d757fa700 1 -- 172.21.15.123:0/356139753 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d6c007e70 con 0x7f6d780f5410 2024-01-13T11:50:56.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.889+0000 7f6d757fa700 1 -- 172.21.15.123:0/356139753 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d6c014ea0 con 0x7f6d780f5410 2024-01-13T11:50:56.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.889+0000 7f6d7f050700 1 -- 172.21.15.123:0/356139753 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d78105d20 con 0x7f6d780f5410 2024-01-13T11:50:56.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.889+0000 7f6d7f050700 1 -- 172.21.15.123:0/356139753 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d781061b0 con 0x7f6d780f5410 2024-01-13T11:50:56.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.890+0000 7f6d757fa700 1 -- 172.21.15.123:0/356139753 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d6c025070 con 0x7f6d780f5410 2024-01-13T11:50:56.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.890+0000 7f6d7f050700 1 -- 172.21.15.123:0/356139753 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d78045ad0 con 0x7f6d780f5410 2024-01-13T11:50:56.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.890+0000 7f6d757fa700 1 --2- 172.21.15.123:0/356139753 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6d6802ffb0 0x7f6d68032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:56.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.890+0000 7f6d757fa700 1 -- 172.21.15.123:0/356139753 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d6c040980 con 0x7f6d780f5410 2024-01-13T11:50:56.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.891+0000 7f6d77fff700 1 --2- 172.21.15.123:0/356139753 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6d6802ffb0 0x7f6d68032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:56.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.891+0000 7f6d77fff700 1 --2- 172.21.15.123:0/356139753 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6d6802ffb0 0x7f6d68032470 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f6d640096a0 tx=0x7f6d64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:56.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:56.895+0000 7f6d757fa700 1 -- 172.21.15.123:0/356139753 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d6c0079c0 con 0x7f6d780f5410 2024-01-13T11:50:57.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:57.103+0000 7f6d7f050700 1 -- 172.21.15.123:0/356139753 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d780fae60 con 0x7f6d780f5410 2024-01-13T11:50:57.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:57.104+0000 7f6d757fa700 1 -- 172.21.15.123:0/356139753 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6d6c00f300 con 0x7f6d780f5410 2024-01-13T11:50:57.093 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:50:57.093 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:50:57.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:57.105+0000 7f6d7f050700 1 -- 172.21.15.123:0/356139753 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6d6802ffb0 msgr2=0x7f6d68032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:57.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:57.105+0000 7f6d7f050700 1 --2- 172.21.15.123:0/356139753 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6d6802ffb0 0x7f6d68032470 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f6d640096a0 tx=0x7f6d64006b40).stop 2024-01-13T11:50:57.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:57.105+0000 7f6d7f050700 1 -- 172.21.15.123:0/356139753 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d780f5410 msgr2=0x7f6d78105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:57.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:57.105+0000 7f6d7f050700 1 --2- 172.21.15.123:0/356139753 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d780f5410 0x7f6d78105630 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f6d6c014000 tx=0x7f6d6c00b040).stop 2024-01-13T11:50:57.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:57.105+0000 7f6d7f050700 1 -- 172.21.15.123:0/356139753 shutdown_connections 2024-01-13T11:50:57.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:57.105+0000 7f6d7f050700 1 --2- 172.21.15.123:0/356139753 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6d6802ffb0 0x7f6d68032470 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:57.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:57.105+0000 7f6d7f050700 1 --2- 172.21.15.123:0/356139753 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d780f5410 0x7f6d78105630 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:57.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:57.105+0000 7f6d7f050700 1 -- 172.21.15.123:0/356139753 >> 172.21.15.123:0/356139753 conn(0x7f6d780f0b70 msgr2=0x7f6d780f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:57.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:57.105+0000 7f6d7f050700 1 -- 172.21.15.123:0/356139753 shutdown_connections 2024-01-13T11:50:57.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:57.105+0000 7f6d7f050700 1 -- 172.21.15.123:0/356139753 wait complete. 2024-01-13T11:50:57.096 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:50:57.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:55.511505+0000 mgr.smithi059.hmyldj (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:57.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:57.104957+0000 mon.smithi059 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.123:0/356139753' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:50:58.472 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:50:58.472 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:50:59.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.583+0000 7fde873f7700 1 -- 172.21.15.123:0/361518491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde800f41a0 msgr2=0x7fde800f45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:59.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.583+0000 7fde873f7700 1 --2- 172.21.15.123:0/361518491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde800f41a0 0x7fde800f45c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fde7c009fb0 tx=0x7fde7c009b30).stop 2024-01-13T11:50:59.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.583+0000 7fde873f7700 1 -- 172.21.15.123:0/361518491 shutdown_connections 2024-01-13T11:50:59.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.583+0000 7fde873f7700 1 --2- 172.21.15.123:0/361518491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde800f41a0 0x7fde800f45c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:59.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.583+0000 7fde873f7700 1 -- 172.21.15.123:0/361518491 >> 172.21.15.123:0/361518491 conn(0x7fde800efe40 msgr2=0x7fde800f2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:59.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.583+0000 7fde873f7700 1 -- 172.21.15.123:0/361518491 shutdown_connections 2024-01-13T11:50:59.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.583+0000 7fde873f7700 1 -- 172.21.15.123:0/361518491 wait complete. 2024-01-13T11:50:59.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.583+0000 7fde873f7700 1 Processor -- start 2024-01-13T11:50:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.583+0000 7fde873f7700 1 -- start start 2024-01-13T11:50:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.583+0000 7fde873f7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde800f41a0 0x7fde800f77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.583+0000 7fde873f7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde800f5c60 con 0x7fde800f41a0 2024-01-13T11:50:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.584+0000 7fde85193700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde800f41a0 0x7fde800f77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.584+0000 7fde85193700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde800f41a0 0x7fde800f77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:41432/0 (socket says 172.21.15.123:41432) 2024-01-13T11:50:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.584+0000 7fde85193700 1 -- 172.21.15.123:0/2663166653 learned_addr learned my addr 172.21.15.123:0/2663166653 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:50:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.584+0000 7fde85193700 1 -- 172.21.15.123:0/2663166653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7c005040 con 0x7fde800f41a0 2024-01-13T11:50:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.584+0000 7fde85193700 1 --2- 172.21.15.123:0/2663166653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde800f41a0 0x7fde800f77e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fde7c013510 tx=0x7fde7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.584+0000 7fde75ffb700 1 -- 172.21.15.123:0/2663166653 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde7c009320 con 0x7fde800f41a0 2024-01-13T11:50:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.584+0000 7fde873f7700 1 -- 172.21.15.123:0/2663166653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde800f5e00 con 0x7fde800f41a0 2024-01-13T11:50:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.584+0000 7fde75ffb700 1 -- 172.21.15.123:0/2663166653 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde7c013f00 con 0x7fde800f41a0 2024-01-13T11:50:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.584+0000 7fde75ffb700 1 -- 172.21.15.123:0/2663166653 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde7c014dc0 con 0x7fde800f41a0 2024-01-13T11:50:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.585+0000 7fde873f7700 1 -- 172.21.15.123:0/2663166653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde800f6290 con 0x7fde800f41a0 2024-01-13T11:50:59.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.585+0000 7fde75ffb700 1 -- 172.21.15.123:0/2663166653 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fde7c0095f0 con 0x7fde800f41a0 2024-01-13T11:50:59.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.585+0000 7fde873f7700 1 -- 172.21.15.123:0/2663166653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde80059100 con 0x7fde800f41a0 2024-01-13T11:50:59.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.585+0000 7fde75ffb700 1 --2- 172.21.15.123:0/2663166653 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fde6c02ff60 0x7fde6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:50:59.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.585+0000 7fde75ffb700 1 -- 172.21.15.123:0/2663166653 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fde7c041100 con 0x7fde800f41a0 2024-01-13T11:50:59.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.586+0000 7fde84992700 1 --2- 172.21.15.123:0/2663166653 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fde6c02ff60 0x7fde6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:50:59.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.586+0000 7fde84992700 1 --2- 172.21.15.123:0/2663166653 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fde6c02ff60 0x7fde6c032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fde700096a0 tx=0x7fde70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:50:59.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.590+0000 7fde75ffb700 1 -- 172.21.15.123:0/2663166653 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fde7c014420 con 0x7fde800f41a0 2024-01-13T11:50:59.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:50:59 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:57.511862+0000 mgr.smithi059.hmyldj (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:59.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.798+0000 7fde873f7700 1 -- 172.21.15.123:0/2663166653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde800fbf40 con 0x7fde800f41a0 2024-01-13T11:50:59.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.799+0000 7fde75ffb700 1 -- 172.21.15.123:0/2663166653 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fde7c010230 con 0x7fde800f41a0 2024-01-13T11:50:59.788 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:50:59.788 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:50:59.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.800+0000 7fde873f7700 1 -- 172.21.15.123:0/2663166653 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fde6c02ff60 msgr2=0x7fde6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:59.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.800+0000 7fde873f7700 1 --2- 172.21.15.123:0/2663166653 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fde6c02ff60 0x7fde6c032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fde700096a0 tx=0x7fde70006b40).stop 2024-01-13T11:50:59.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.800+0000 7fde873f7700 1 -- 172.21.15.123:0/2663166653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde800f41a0 msgr2=0x7fde800f77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:50:59.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.800+0000 7fde873f7700 1 --2- 172.21.15.123:0/2663166653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde800f41a0 0x7fde800f77e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fde7c013510 tx=0x7fde7c00b040).stop 2024-01-13T11:50:59.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.800+0000 7fde873f7700 1 -- 172.21.15.123:0/2663166653 shutdown_connections 2024-01-13T11:50:59.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.800+0000 7fde873f7700 1 --2- 172.21.15.123:0/2663166653 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fde6c02ff60 0x7fde6c032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:59.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.800+0000 7fde873f7700 1 --2- 172.21.15.123:0/2663166653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde800f41a0 0x7fde800f77e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:50:59.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.800+0000 7fde873f7700 1 -- 172.21.15.123:0/2663166653 >> 172.21.15.123:0/2663166653 conn(0x7fde800efe40 msgr2=0x7fde800f0950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:50:59.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.800+0000 7fde873f7700 1 -- 172.21.15.123:0/2663166653 shutdown_connections 2024-01-13T11:50:59.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:50:59.800+0000 7fde873f7700 1 -- 172.21.15.123:0/2663166653 wait complete. 2024-01-13T11:50:59.791 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:00.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:00 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:50:59.799950+0000 mon.smithi059 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.123:0/2663166653' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:01.171 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:01.172 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:01.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:01 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:50:59.512157+0000 mgr.smithi059.hmyldj (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:02.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.333+0000 7faa08e98700 1 -- 172.21.15.123:0/1362711371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa040f47c0 msgr2=0x7faa040f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:02.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.333+0000 7faa08e98700 1 --2- 172.21.15.123:0/1362711371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa040f47c0 0x7faa040f4be0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fa9f4009fb0 tx=0x7fa9f4009b30).stop 2024-01-13T11:51:02.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.333+0000 7faa08e98700 1 -- 172.21.15.123:0/1362711371 shutdown_connections 2024-01-13T11:51:02.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.333+0000 7faa08e98700 1 --2- 172.21.15.123:0/1362711371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa040f47c0 0x7faa040f4be0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:02.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.333+0000 7faa08e98700 1 -- 172.21.15.123:0/1362711371 >> 172.21.15.123:0/1362711371 conn(0x7faa040f02e0 msgr2=0x7faa040f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:02.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.333+0000 7faa08e98700 1 -- 172.21.15.123:0/1362711371 shutdown_connections 2024-01-13T11:51:02.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.333+0000 7faa08e98700 1 -- 172.21.15.123:0/1362711371 wait complete. 2024-01-13T11:51:02.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.333+0000 7faa08e98700 1 Processor -- start 2024-01-13T11:51:02.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.333+0000 7faa08e98700 1 -- start start 2024-01-13T11:51:02.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.333+0000 7faa08e98700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa040f47c0 0x7faa04108900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:02.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.333+0000 7faa08e98700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa04108df0 con 0x7faa040f47c0 2024-01-13T11:51:02.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.334+0000 7faa0259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa040f47c0 0x7faa04108900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:02.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.334+0000 7faa0259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa040f47c0 0x7faa04108900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:41446/0 (socket says 172.21.15.123:41446) 2024-01-13T11:51:02.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.334+0000 7faa0259c700 1 -- 172.21.15.123:0/545575104 learned_addr learned my addr 172.21.15.123:0/545575104 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:02.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.334+0000 7faa0259c700 1 -- 172.21.15.123:0/545575104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f4005040 con 0x7faa040f47c0 2024-01-13T11:51:02.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.334+0000 7faa0259c700 1 --2- 172.21.15.123:0/545575104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa040f47c0 0x7faa04108900 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fa9f4013510 tx=0x7fa9f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:02.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.334+0000 7fa9f2ffd700 1 -- 172.21.15.123:0/545575104 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9f4009320 con 0x7faa040f47c0 2024-01-13T11:51:02.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.334+0000 7fa9f2ffd700 1 -- 172.21.15.123:0/545575104 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa9f4013f00 con 0x7faa040f47c0 2024-01-13T11:51:02.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.334+0000 7faa08e98700 1 -- 172.21.15.123:0/545575104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa04108ff0 con 0x7faa040f47c0 2024-01-13T11:51:02.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.334+0000 7fa9f2ffd700 1 -- 172.21.15.123:0/545575104 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9f4014d90 con 0x7faa040f47c0 2024-01-13T11:51:02.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.334+0000 7faa08e98700 1 -- 172.21.15.123:0/545575104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa04109480 con 0x7faa040f47c0 2024-01-13T11:51:02.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.335+0000 7fa9f2ffd700 1 -- 172.21.15.123:0/545575104 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9f40263a0 con 0x7faa040f47c0 2024-01-13T11:51:02.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.335+0000 7fa9f2ffd700 1 --2- 172.21.15.123:0/545575104 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa9ec02ff60 0x7fa9ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:02.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.335+0000 7faa08e98700 1 -- 172.21.15.123:0/545575104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa04045ad0 con 0x7faa040f47c0 2024-01-13T11:51:02.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.335+0000 7fa9f2ffd700 1 -- 172.21.15.123:0/545575104 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa9f4040fe0 con 0x7faa040f47c0 2024-01-13T11:51:02.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.335+0000 7faa01d9b700 1 --2- 172.21.15.123:0/545575104 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa9ec02ff60 0x7fa9ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:02.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.336+0000 7faa01d9b700 1 --2- 172.21.15.123:0/545575104 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa9ec02ff60 0x7fa9ec032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fa9f80096a0 tx=0x7fa9f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:02.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.340+0000 7fa9f2ffd700 1 -- 172.21.15.123:0/545575104 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9f4012920 con 0x7faa040f47c0 2024-01-13T11:51:02.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.545+0000 7faa08e98700 1 -- 172.21.15.123:0/545575104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa0410a2d0 con 0x7faa040f47c0 2024-01-13T11:51:02.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.546+0000 7fa9f2ffd700 1 -- 172.21.15.123:0/545575104 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa9f400f340 con 0x7faa040f47c0 2024-01-13T11:51:02.535 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:02.535 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:02.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.547+0000 7faa08e98700 1 -- 172.21.15.123:0/545575104 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa9ec02ff60 msgr2=0x7fa9ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:02.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.547+0000 7faa08e98700 1 --2- 172.21.15.123:0/545575104 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa9ec02ff60 0x7fa9ec032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fa9f80096a0 tx=0x7fa9f8006b40).stop 2024-01-13T11:51:02.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.547+0000 7faa08e98700 1 -- 172.21.15.123:0/545575104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa040f47c0 msgr2=0x7faa04108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:02.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.547+0000 7faa08e98700 1 --2- 172.21.15.123:0/545575104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa040f47c0 0x7faa04108900 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fa9f4013510 tx=0x7fa9f400b040).stop 2024-01-13T11:51:02.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.547+0000 7faa08e98700 1 -- 172.21.15.123:0/545575104 shutdown_connections 2024-01-13T11:51:02.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.547+0000 7faa08e98700 1 --2- 172.21.15.123:0/545575104 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa9ec02ff60 0x7fa9ec032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:02.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.547+0000 7faa08e98700 1 --2- 172.21.15.123:0/545575104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa040f47c0 0x7faa04108900 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:02.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.547+0000 7faa08e98700 1 -- 172.21.15.123:0/545575104 >> 172.21.15.123:0/545575104 conn(0x7faa040f02e0 msgr2=0x7faa040f0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:02.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.547+0000 7faa08e98700 1 -- 172.21.15.123:0/545575104 shutdown_connections 2024-01-13T11:51:02.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:02.547+0000 7faa08e98700 1 -- 172.21.15.123:0/545575104 wait complete. 2024-01-13T11:51:02.538 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:03.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:03 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:01.512491+0000 mgr.smithi059.hmyldj (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:03.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:03 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:02.546753+0000 mon.smithi059 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.123:0/545575104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:03.917 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:03.917 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:05.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.057+0000 7f430359e700 1 -- 172.21.15.123:0/4213552849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43040f51a0 msgr2=0x7f43040f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:05.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.057+0000 7f430359e700 1 --2- 172.21.15.123:0/4213552849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43040f51a0 0x7f43040f55c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f42f8009fb0 tx=0x7f42f8009b30).stop 2024-01-13T11:51:05.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.057+0000 7f430359e700 1 -- 172.21.15.123:0/4213552849 shutdown_connections 2024-01-13T11:51:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.057+0000 7f430359e700 1 --2- 172.21.15.123:0/4213552849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43040f51a0 0x7f43040f55c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.057+0000 7f430359e700 1 -- 172.21.15.123:0/4213552849 >> 172.21.15.123:0/4213552849 conn(0x7f43040f09a0 msgr2=0x7f43040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.057+0000 7f430359e700 1 -- 172.21.15.123:0/4213552849 shutdown_connections 2024-01-13T11:51:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.057+0000 7f430359e700 1 -- 172.21.15.123:0/4213552849 wait complete. 2024-01-13T11:51:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.057+0000 7f430359e700 1 Processor -- start 2024-01-13T11:51:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.057+0000 7f430359e700 1 -- start start 2024-01-13T11:51:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.058+0000 7f430359e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43040f51a0 0x7f43041091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.058+0000 7f430359e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4304060240 con 0x7f43040f51a0 2024-01-13T11:51:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.058+0000 7f430259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43040f51a0 0x7f43041091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.058+0000 7f430259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43040f51a0 0x7f43041091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:41460/0 (socket says 172.21.15.123:41460) 2024-01-13T11:51:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.058+0000 7f430259c700 1 -- 172.21.15.123:0/3790755179 learned_addr learned my addr 172.21.15.123:0/3790755179 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.058+0000 7f430259c700 1 -- 172.21.15.123:0/3790755179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f8005040 con 0x7f43040f51a0 2024-01-13T11:51:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.058+0000 7f430259c700 1 --2- 172.21.15.123:0/3790755179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43040f51a0 0x7f43041091b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f42f8013510 tx=0x7f42f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:05.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.058+0000 7f42f2ffd700 1 -- 172.21.15.123:0/3790755179 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42f8009320 con 0x7f43040f51a0 2024-01-13T11:51:05.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.059+0000 7f430359e700 1 -- 172.21.15.123:0/3790755179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4304060440 con 0x7f43040f51a0 2024-01-13T11:51:05.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.059+0000 7f42f2ffd700 1 -- 172.21.15.123:0/3790755179 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42f8013f00 con 0x7f43040f51a0 2024-01-13T11:51:05.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.059+0000 7f42f2ffd700 1 -- 172.21.15.123:0/3790755179 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42f8014dc0 con 0x7f43040f51a0 2024-01-13T11:51:05.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.059+0000 7f430359e700 1 -- 172.21.15.123:0/3790755179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4304109860 con 0x7f43040f51a0 2024-01-13T11:51:05.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.059+0000 7f42f2ffd700 1 -- 172.21.15.123:0/3790755179 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42f80095f0 con 0x7f43040f51a0 2024-01-13T11:51:05.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.059+0000 7f430359e700 1 -- 172.21.15.123:0/3790755179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4304045ad0 con 0x7f43040f51a0 2024-01-13T11:51:05.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.059+0000 7f42f2ffd700 1 --2- 172.21.15.123:0/3790755179 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f42ec02ff60 0x7f42ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:05.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.059+0000 7f42f2ffd700 1 -- 172.21.15.123:0/3790755179 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f42f8041000 con 0x7f43040f51a0 2024-01-13T11:51:05.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.060+0000 7f4301d9b700 1 --2- 172.21.15.123:0/3790755179 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f42ec02ff60 0x7f42ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:05.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.060+0000 7f4301d9b700 1 --2- 172.21.15.123:0/3790755179 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f42ec02ff60 0x7f42ec032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f42f40096a0 tx=0x7f42f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:05.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.065+0000 7f42f2ffd700 1 -- 172.21.15.123:0/3790755179 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42f8014420 con 0x7f43040f51a0 2024-01-13T11:51:05.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.272+0000 7f430359e700 1 -- 172.21.15.123:0/3790755179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f430410a080 con 0x7f43040f51a0 2024-01-13T11:51:05.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.272+0000 7f42f2ffd700 1 -- 172.21.15.123:0/3790755179 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f42f8010230 con 0x7f43040f51a0 2024-01-13T11:51:05.262 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:05.262 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:05.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.274+0000 7f430359e700 1 -- 172.21.15.123:0/3790755179 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f42ec02ff60 msgr2=0x7f42ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:05.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.274+0000 7f430359e700 1 --2- 172.21.15.123:0/3790755179 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f42ec02ff60 0x7f42ec032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f42f40096a0 tx=0x7f42f4006b40).stop 2024-01-13T11:51:05.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.274+0000 7f430359e700 1 -- 172.21.15.123:0/3790755179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43040f51a0 msgr2=0x7f43041091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:05.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.274+0000 7f430359e700 1 --2- 172.21.15.123:0/3790755179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43040f51a0 0x7f43041091b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f42f8013510 tx=0x7f42f800b040).stop 2024-01-13T11:51:05.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.274+0000 7f430359e700 1 -- 172.21.15.123:0/3790755179 shutdown_connections 2024-01-13T11:51:05.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.274+0000 7f430359e700 1 --2- 172.21.15.123:0/3790755179 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f42ec02ff60 0x7f42ec032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:05.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.274+0000 7f430359e700 1 --2- 172.21.15.123:0/3790755179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43040f51a0 0x7f43041091b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:05.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.274+0000 7f430359e700 1 -- 172.21.15.123:0/3790755179 >> 172.21.15.123:0/3790755179 conn(0x7f43040f09a0 msgr2=0x7f43040f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:05.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.274+0000 7f430359e700 1 -- 172.21.15.123:0/3790755179 shutdown_connections 2024-01-13T11:51:05.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:05.274+0000 7f430359e700 1 -- 172.21.15.123:0/3790755179 wait complete. 2024-01-13T11:51:05.265 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:05.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:05 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:03.512984+0000 mgr.smithi059.hmyldj (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:06.631 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:06.632 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:06.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:06 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:05.273704+0000 mon.smithi059 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.123:0/3790755179' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:07.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:05.513287+0000 mgr.smithi059.hmyldj (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:07.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.740+0000 7fec92f84700 1 -- 172.21.15.123:0/656937206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec8c0f5410 msgr2=0x7fec8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:07.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.740+0000 7fec92f84700 1 --2- 172.21.15.123:0/656937206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec8c0f5410 0x7fec8c0f5830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fec80009fb0 tx=0x7fec80009b30).stop 2024-01-13T11:51:07.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.740+0000 7fec92f84700 1 -- 172.21.15.123:0/656937206 shutdown_connections 2024-01-13T11:51:07.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.740+0000 7fec92f84700 1 --2- 172.21.15.123:0/656937206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec8c0f5410 0x7fec8c0f5830 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:07.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.740+0000 7fec92f84700 1 -- 172.21.15.123:0/656937206 >> 172.21.15.123:0/656937206 conn(0x7fec8c0f0b70 msgr2=0x7fec8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:07.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.740+0000 7fec92f84700 1 -- 172.21.15.123:0/656937206 shutdown_connections 2024-01-13T11:51:07.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.740+0000 7fec92f84700 1 -- 172.21.15.123:0/656937206 wait complete. 2024-01-13T11:51:07.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.740+0000 7fec92f84700 1 Processor -- start 2024-01-13T11:51:07.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.740+0000 7fec92f84700 1 -- start start 2024-01-13T11:51:07.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.741+0000 7fec92f84700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec8c0f5410 0x7fec8c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:07.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.741+0000 7fec92f84700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c106bd0 con 0x7fec8c0f5410 2024-01-13T11:51:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.741+0000 7fec90d20700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec8c0f5410 0x7fec8c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.741+0000 7fec90d20700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec8c0f5410 0x7fec8c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:52024/0 (socket says 172.21.15.123:52024) 2024-01-13T11:51:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.741+0000 7fec90d20700 1 -- 172.21.15.123:0/3056960808 learned_addr learned my addr 172.21.15.123:0/3056960808 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.741+0000 7fec90d20700 1 -- 172.21.15.123:0/3056960808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec80005040 con 0x7fec8c0f5410 2024-01-13T11:51:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.741+0000 7fec90d20700 1 --2- 172.21.15.123:0/3056960808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec8c0f5410 0x7fec8c1066e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fec80014000 tx=0x7fec8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.742+0000 7fec897fa700 1 -- 172.21.15.123:0/3056960808 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec800095f0 con 0x7fec8c0f5410 2024-01-13T11:51:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.742+0000 7fec92f84700 1 -- 172.21.15.123:0/3056960808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec8c106dd0 con 0x7fec8c0f5410 2024-01-13T11:51:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.742+0000 7fec897fa700 1 -- 172.21.15.123:0/3056960808 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec80007e70 con 0x7fec8c0f5410 2024-01-13T11:51:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.742+0000 7fec897fa700 1 -- 172.21.15.123:0/3056960808 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec80014ea0 con 0x7fec8c0f5410 2024-01-13T11:51:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.742+0000 7fec92f84700 1 -- 172.21.15.123:0/3056960808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8c107260 con 0x7fec8c0f5410 2024-01-13T11:51:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.742+0000 7fec897fa700 1 -- 172.21.15.123:0/3056960808 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec80025070 con 0x7fec8c0f5410 2024-01-13T11:51:07.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.742+0000 7fec92f84700 1 -- 172.21.15.123:0/3056960808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec8c045ad0 con 0x7fec8c0f5410 2024-01-13T11:51:07.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.743+0000 7fec897fa700 1 --2- 172.21.15.123:0/3056960808 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fec7c02ff60 0x7fec7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:07.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.743+0000 7fec897fa700 1 -- 172.21.15.123:0/3056960808 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec80040a00 con 0x7fec8c0f5410 2024-01-13T11:51:07.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.743+0000 7fec8bfff700 1 --2- 172.21.15.123:0/3056960808 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fec7c02ff60 0x7fec7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:07.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.743+0000 7fec8bfff700 1 --2- 172.21.15.123:0/3056960808 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fec7c02ff60 0x7fec7c032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fec780096a0 tx=0x7fec78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:07.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.747+0000 7fec897fa700 1 -- 172.21.15.123:0/3056960808 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec800079c0 con 0x7fec8c0f5410 2024-01-13T11:51:07.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.958+0000 7fec92f84700 1 -- 172.21.15.123:0/3056960808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec8c108110 con 0x7fec8c0f5410 2024-01-13T11:51:07.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.958+0000 7fec897fa700 1 -- 172.21.15.123:0/3056960808 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fec8000f300 con 0x7fec8c0f5410 2024-01-13T11:51:07.947 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:07.947 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:07.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.960+0000 7fec92f84700 1 -- 172.21.15.123:0/3056960808 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fec7c02ff60 msgr2=0x7fec7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:07.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.960+0000 7fec92f84700 1 --2- 172.21.15.123:0/3056960808 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fec7c02ff60 0x7fec7c032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fec780096a0 tx=0x7fec78006b40).stop 2024-01-13T11:51:07.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.960+0000 7fec92f84700 1 -- 172.21.15.123:0/3056960808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec8c0f5410 msgr2=0x7fec8c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:07.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.960+0000 7fec92f84700 1 --2- 172.21.15.123:0/3056960808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec8c0f5410 0x7fec8c1066e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fec80014000 tx=0x7fec8000b040).stop 2024-01-13T11:51:07.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.960+0000 7fec92f84700 1 -- 172.21.15.123:0/3056960808 shutdown_connections 2024-01-13T11:51:07.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.960+0000 7fec92f84700 1 --2- 172.21.15.123:0/3056960808 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fec7c02ff60 0x7fec7c032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:07.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.960+0000 7fec92f84700 1 --2- 172.21.15.123:0/3056960808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec8c0f5410 0x7fec8c1066e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:07.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.960+0000 7fec92f84700 1 -- 172.21.15.123:0/3056960808 >> 172.21.15.123:0/3056960808 conn(0x7fec8c0f0b70 msgr2=0x7fec8c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:07.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.960+0000 7fec92f84700 1 -- 172.21.15.123:0/3056960808 shutdown_connections 2024-01-13T11:51:07.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:07.960+0000 7fec92f84700 1 -- 172.21.15.123:0/3056960808 wait complete. 2024-01-13T11:51:07.950 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:08.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:08 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:07.958566+0000 mon.smithi059 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.123:0/3056960808' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:09.329 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:09.329 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:09.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:09 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:07.513596+0000 mgr.smithi059.hmyldj (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:10.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.432+0000 7f76e9586700 1 -- 172.21.15.123:0/155458865 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76e40f62a0 msgr2=0x7f76e40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:10.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.432+0000 7f76e9586700 1 --2- 172.21.15.123:0/155458865 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76e40f62a0 0x7f76e40f66c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f76d4009fb0 tx=0x7f76d4009b30).stop 2024-01-13T11:51:10.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.432+0000 7f76e9586700 1 -- 172.21.15.123:0/155458865 shutdown_connections 2024-01-13T11:51:10.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.432+0000 7f76e9586700 1 --2- 172.21.15.123:0/155458865 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76e40f62a0 0x7f76e40f66c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:10.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.432+0000 7f76e9586700 1 -- 172.21.15.123:0/155458865 >> 172.21.15.123:0/155458865 conn(0x7f76e40f1aa0 msgr2=0x7f76e40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:10.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.432+0000 7f76e9586700 1 -- 172.21.15.123:0/155458865 shutdown_connections 2024-01-13T11:51:10.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.432+0000 7f76e9586700 1 -- 172.21.15.123:0/155458865 wait complete. 2024-01-13T11:51:10.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.432+0000 7f76e9586700 1 Processor -- start 2024-01-13T11:51:10.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.432+0000 7f76e9586700 1 -- start start 2024-01-13T11:51:10.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.433+0000 7f76e9586700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76e40f62a0 0x7f76e41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:10.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.433+0000 7f76e9586700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e41069a0 con 0x7f76e40f62a0 2024-01-13T11:51:10.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.433+0000 7f76e3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76e40f62a0 0x7f76e41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:10.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.433+0000 7f76e3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76e40f62a0 0x7f76e41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:52056/0 (socket says 172.21.15.123:52056) 2024-01-13T11:51:10.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.433+0000 7f76e3fff700 1 -- 172.21.15.123:0/2386514220 learned_addr learned my addr 172.21.15.123:0/2386514220 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:10.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.433+0000 7f76e3fff700 1 -- 172.21.15.123:0/2386514220 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d4005040 con 0x7f76e40f62a0 2024-01-13T11:51:10.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.433+0000 7f76e3fff700 1 --2- 172.21.15.123:0/2386514220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76e40f62a0 0x7f76e41064b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f76d4013510 tx=0x7f76d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:10.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.433+0000 7f76e0ff9700 1 -- 172.21.15.123:0/2386514220 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76d4009320 con 0x7f76e40f62a0 2024-01-13T11:51:10.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.433+0000 7f76e9586700 1 -- 172.21.15.123:0/2386514220 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e4106ba0 con 0x7f76e40f62a0 2024-01-13T11:51:10.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.433+0000 7f76e0ff9700 1 -- 172.21.15.123:0/2386514220 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76d4013f00 con 0x7f76e40f62a0 2024-01-13T11:51:10.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.433+0000 7f76e0ff9700 1 -- 172.21.15.123:0/2386514220 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76d4014dc0 con 0x7f76e40f62a0 2024-01-13T11:51:10.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.433+0000 7f76e9586700 1 -- 172.21.15.123:0/2386514220 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e4107030 con 0x7f76e40f62a0 2024-01-13T11:51:10.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.434+0000 7f76e9586700 1 -- 172.21.15.123:0/2386514220 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e4045ad0 con 0x7f76e40f62a0 2024-01-13T11:51:10.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.434+0000 7f76e0ff9700 1 -- 172.21.15.123:0/2386514220 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f76d4014420 con 0x7f76e40f62a0 2024-01-13T11:51:10.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.435+0000 7f76e0ff9700 1 --2- 172.21.15.123:0/2386514220 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f76cc02ff60 0x7f76cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:10.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.435+0000 7f76e0ff9700 1 -- 172.21.15.123:0/2386514220 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f76d4041050 con 0x7f76e40f62a0 2024-01-13T11:51:10.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.435+0000 7f76e37fe700 1 --2- 172.21.15.123:0/2386514220 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f76cc02ff60 0x7f76cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:10.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.435+0000 7f76e37fe700 1 --2- 172.21.15.123:0/2386514220 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f76cc02ff60 0x7f76cc032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f76d80096a0 tx=0x7f76d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:10.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.439+0000 7f76e0ff9700 1 -- 172.21.15.123:0/2386514220 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76d40095f0 con 0x7f76e40f62a0 2024-01-13T11:51:10.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.645+0000 7f76e9586700 1 -- 172.21.15.123:0/2386514220 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76e4107ee0 con 0x7f76e40f62a0 2024-01-13T11:51:10.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.646+0000 7f76e0ff9700 1 -- 172.21.15.123:0/2386514220 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f76d401cb10 con 0x7f76e40f62a0 2024-01-13T11:51:10.634 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:10.634 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:10.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.647+0000 7f76e9586700 1 -- 172.21.15.123:0/2386514220 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f76cc02ff60 msgr2=0x7f76cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:10.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.647+0000 7f76e9586700 1 --2- 172.21.15.123:0/2386514220 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f76cc02ff60 0x7f76cc032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f76d80096a0 tx=0x7f76d8006b40).stop 2024-01-13T11:51:10.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.647+0000 7f76e9586700 1 -- 172.21.15.123:0/2386514220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76e40f62a0 msgr2=0x7f76e41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:10.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.647+0000 7f76e9586700 1 --2- 172.21.15.123:0/2386514220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76e40f62a0 0x7f76e41064b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f76d4013510 tx=0x7f76d400b040).stop 2024-01-13T11:51:10.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.647+0000 7f76e9586700 1 -- 172.21.15.123:0/2386514220 shutdown_connections 2024-01-13T11:51:10.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.647+0000 7f76e9586700 1 --2- 172.21.15.123:0/2386514220 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f76cc02ff60 0x7f76cc032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:10.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.647+0000 7f76e9586700 1 --2- 172.21.15.123:0/2386514220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76e40f62a0 0x7f76e41064b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:10.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.647+0000 7f76e9586700 1 -- 172.21.15.123:0/2386514220 >> 172.21.15.123:0/2386514220 conn(0x7f76e40f1aa0 msgr2=0x7f76e40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:10.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.648+0000 7f76e9586700 1 -- 172.21.15.123:0/2386514220 shutdown_connections 2024-01-13T11:51:10.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:10.648+0000 7f76e9586700 1 -- 172.21.15.123:0/2386514220 wait complete. 2024-01-13T11:51:10.637 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:11.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:11 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:09.513828+0000 mgr.smithi059.hmyldj (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:11.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:11 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:10.646127+0000 mon.smithi059 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.123:0/2386514220' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:12.038 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:12.039 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:12.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:12 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:11.514063+0000 mgr.smithi059.hmyldj (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:13.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.215+0000 7f1b76f75700 1 -- 172.21.15.123:0/3634052713 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b700f6510 msgr2=0x7f1b700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:13.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.215+0000 7f1b76f75700 1 --2- 172.21.15.123:0/3634052713 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b700f6510 0x7f1b700f6930 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f1b64009fb0 tx=0x7f1b64009b30).stop 2024-01-13T11:51:13.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.215+0000 7f1b76f75700 1 -- 172.21.15.123:0/3634052713 shutdown_connections 2024-01-13T11:51:13.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.215+0000 7f1b76f75700 1 --2- 172.21.15.123:0/3634052713 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b700f6510 0x7f1b700f6930 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:13.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.215+0000 7f1b76f75700 1 -- 172.21.15.123:0/3634052713 >> 172.21.15.123:0/3634052713 conn(0x7f1b700f1c70 msgr2=0x7f1b700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:13.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.215+0000 7f1b76f75700 1 -- 172.21.15.123:0/3634052713 shutdown_connections 2024-01-13T11:51:13.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.215+0000 7f1b76f75700 1 -- 172.21.15.123:0/3634052713 wait complete. 2024-01-13T11:51:13.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.216+0000 7f1b76f75700 1 Processor -- start 2024-01-13T11:51:13.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.216+0000 7f1b76f75700 1 -- start start 2024-01-13T11:51:13.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.216+0000 7f1b76f75700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b700f6510 0x7f1b70106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:13.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.216+0000 7f1b76f75700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b70106c20 con 0x7f1b700f6510 2024-01-13T11:51:13.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.216+0000 7f1b74d11700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b700f6510 0x7f1b70106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:13.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.216+0000 7f1b74d11700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b700f6510 0x7f1b70106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:52074/0 (socket says 172.21.15.123:52074) 2024-01-13T11:51:13.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.216+0000 7f1b74d11700 1 -- 172.21.15.123:0/4274382506 learned_addr learned my addr 172.21.15.123:0/4274382506 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:13.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.217+0000 7f1b74d11700 1 -- 172.21.15.123:0/4274382506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b64005040 con 0x7f1b700f6510 2024-01-13T11:51:13.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.217+0000 7f1b74d11700 1 --2- 172.21.15.123:0/4274382506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b700f6510 0x7f1b70106730 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f1b64013510 tx=0x7f1b6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:13.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.217+0000 7f1b6d7fa700 1 -- 172.21.15.123:0/4274382506 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b64009320 con 0x7f1b700f6510 2024-01-13T11:51:13.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.217+0000 7f1b76f75700 1 -- 172.21.15.123:0/4274382506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b70106e20 con 0x7f1b700f6510 2024-01-13T11:51:13.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.217+0000 7f1b6d7fa700 1 -- 172.21.15.123:0/4274382506 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b64013f00 con 0x7f1b700f6510 2024-01-13T11:51:13.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.217+0000 7f1b6d7fa700 1 -- 172.21.15.123:0/4274382506 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b64014d90 con 0x7f1b700f6510 2024-01-13T11:51:13.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.217+0000 7f1b76f75700 1 -- 172.21.15.123:0/4274382506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b701072b0 con 0x7f1b700f6510 2024-01-13T11:51:13.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.218+0000 7f1b6d7fa700 1 -- 172.21.15.123:0/4274382506 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b640095f0 con 0x7f1b700f6510 2024-01-13T11:51:13.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.218+0000 7f1b76f75700 1 -- 172.21.15.123:0/4274382506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b70045ad0 con 0x7f1b700f6510 2024-01-13T11:51:13.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.218+0000 7f1b6d7fa700 1 --2- 172.21.15.123:0/4274382506 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1b6002ff60 0x7f1b60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:13.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.218+0000 7f1b6d7fa700 1 -- 172.21.15.123:0/4274382506 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b64040fc0 con 0x7f1b700f6510 2024-01-13T11:51:13.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.218+0000 7f1b6ffff700 1 --2- 172.21.15.123:0/4274382506 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1b6002ff60 0x7f1b60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:13.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.219+0000 7f1b6ffff700 1 --2- 172.21.15.123:0/4274382506 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1b6002ff60 0x7f1b60032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f1b5c0096a0 tx=0x7f1b5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:13.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.223+0000 7f1b6d7fa700 1 -- 172.21.15.123:0/4274382506 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b64014420 con 0x7f1b700f6510 2024-01-13T11:51:13.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.432+0000 7f1b76f75700 1 -- 172.21.15.123:0/4274382506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b700fbf60 con 0x7f1b700f6510 2024-01-13T11:51:13.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.432+0000 7f1b6d7fa700 1 -- 172.21.15.123:0/4274382506 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1b64010230 con 0x7f1b700f6510 2024-01-13T11:51:13.423 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:13.423 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:13.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.434+0000 7f1b76f75700 1 -- 172.21.15.123:0/4274382506 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1b6002ff60 msgr2=0x7f1b60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:13.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.434+0000 7f1b76f75700 1 --2- 172.21.15.123:0/4274382506 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1b6002ff60 0x7f1b60032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f1b5c0096a0 tx=0x7f1b5c006b40).stop 2024-01-13T11:51:13.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.434+0000 7f1b76f75700 1 -- 172.21.15.123:0/4274382506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b700f6510 msgr2=0x7f1b70106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:13.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.434+0000 7f1b76f75700 1 --2- 172.21.15.123:0/4274382506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b700f6510 0x7f1b70106730 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f1b64013510 tx=0x7f1b6400b040).stop 2024-01-13T11:51:13.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.434+0000 7f1b76f75700 1 -- 172.21.15.123:0/4274382506 shutdown_connections 2024-01-13T11:51:13.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.434+0000 7f1b76f75700 1 --2- 172.21.15.123:0/4274382506 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1b6002ff60 0x7f1b60032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:13.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.434+0000 7f1b76f75700 1 --2- 172.21.15.123:0/4274382506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b700f6510 0x7f1b70106730 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:13.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.434+0000 7f1b76f75700 1 -- 172.21.15.123:0/4274382506 >> 172.21.15.123:0/4274382506 conn(0x7f1b700f1c70 msgr2=0x7f1b700f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:13.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.434+0000 7f1b76f75700 1 -- 172.21.15.123:0/4274382506 shutdown_connections 2024-01-13T11:51:13.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:13.434+0000 7f1b76f75700 1 -- 172.21.15.123:0/4274382506 wait complete. 2024-01-13T11:51:13.427 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:13.888 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:13 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:13.433850+0000 mon.smithi059 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.123:0/4274382506' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:14.810 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:14.811 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:14.888 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:13.514358+0000 mgr.smithi059.hmyldj (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:15.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.949+0000 7f5d70440700 1 -- 172.21.15.123:0/2960748079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d680f5410 msgr2=0x7f5d680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:15.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.949+0000 7f5d70440700 1 --2- 172.21.15.123:0/2960748079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d680f5410 0x7f5d680f5830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f5d58009fb0 tx=0x7f5d58009b30).stop 2024-01-13T11:51:15.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.949+0000 7f5d70440700 1 -- 172.21.15.123:0/2960748079 shutdown_connections 2024-01-13T11:51:15.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.949+0000 7f5d70440700 1 --2- 172.21.15.123:0/2960748079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d680f5410 0x7f5d680f5830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:15.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.949+0000 7f5d70440700 1 -- 172.21.15.123:0/2960748079 >> 172.21.15.123:0/2960748079 conn(0x7f5d680f0b70 msgr2=0x7f5d680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:15.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.949+0000 7f5d70440700 1 -- 172.21.15.123:0/2960748079 shutdown_connections 2024-01-13T11:51:15.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.949+0000 7f5d70440700 1 -- 172.21.15.123:0/2960748079 wait complete. 2024-01-13T11:51:15.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.950+0000 7f5d70440700 1 Processor -- start 2024-01-13T11:51:15.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.950+0000 7f5d70440700 1 -- start start 2024-01-13T11:51:15.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.950+0000 7f5d70440700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d680f5410 0x7f5d68105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:15.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.950+0000 7f5d70440700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d68105b20 con 0x7f5d680f5410 2024-01-13T11:51:15.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.951+0000 7f5d6e1dc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d680f5410 0x7f5d68105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:15.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.951+0000 7f5d6e1dc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d680f5410 0x7f5d68105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:52098/0 (socket says 172.21.15.123:52098) 2024-01-13T11:51:15.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.951+0000 7f5d6e1dc700 1 -- 172.21.15.123:0/3420181641 learned_addr learned my addr 172.21.15.123:0/3420181641 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:15.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.951+0000 7f5d6e1dc700 1 -- 172.21.15.123:0/3420181641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d58005040 con 0x7f5d680f5410 2024-01-13T11:51:15.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.951+0000 7f5d6e1dc700 1 --2- 172.21.15.123:0/3420181641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d680f5410 0x7f5d68105630 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f5d58013510 tx=0x7f5d5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:15.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.951+0000 7f5d5effd700 1 -- 172.21.15.123:0/3420181641 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d58009320 con 0x7f5d680f5410 2024-01-13T11:51:15.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.951+0000 7f5d70440700 1 -- 172.21.15.123:0/3420181641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d68105d20 con 0x7f5d680f5410 2024-01-13T11:51:15.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.951+0000 7f5d5effd700 1 -- 172.21.15.123:0/3420181641 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d58013f00 con 0x7f5d680f5410 2024-01-13T11:51:15.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.951+0000 7f5d5effd700 1 -- 172.21.15.123:0/3420181641 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d58014d90 con 0x7f5d680f5410 2024-01-13T11:51:15.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.951+0000 7f5d70440700 1 -- 172.21.15.123:0/3420181641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d681061b0 con 0x7f5d680f5410 2024-01-13T11:51:15.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.952+0000 7f5d5effd700 1 -- 172.21.15.123:0/3420181641 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d580095f0 con 0x7f5d680f5410 2024-01-13T11:51:15.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.952+0000 7f5d70440700 1 -- 172.21.15.123:0/3420181641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d68045ad0 con 0x7f5d680f5410 2024-01-13T11:51:15.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.952+0000 7f5d5effd700 1 --2- 172.21.15.123:0/3420181641 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5d5402ff60 0x7f5d54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:15.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.952+0000 7f5d5effd700 1 -- 172.21.15.123:0/3420181641 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d58041000 con 0x7f5d680f5410 2024-01-13T11:51:15.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.953+0000 7f5d6d9db700 1 --2- 172.21.15.123:0/3420181641 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5d5402ff60 0x7f5d54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:15.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.953+0000 7f5d6d9db700 1 --2- 172.21.15.123:0/3420181641 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5d5402ff60 0x7f5d54032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f5d640096a0 tx=0x7f5d64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:15.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:15.957+0000 7f5d5effd700 1 -- 172.21.15.123:0/3420181641 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d58014420 con 0x7f5d680f5410 2024-01-13T11:51:16.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:16.165+0000 7f5d70440700 1 -- 172.21.15.123:0/3420181641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d680fae60 con 0x7f5d680f5410 2024-01-13T11:51:16.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:16.165+0000 7f5d5effd700 1 -- 172.21.15.123:0/3420181641 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5d5801d7f0 con 0x7f5d680f5410 2024-01-13T11:51:16.155 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:16.155 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:16.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:16.167+0000 7f5d70440700 1 -- 172.21.15.123:0/3420181641 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5d5402ff60 msgr2=0x7f5d54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:16.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:16.167+0000 7f5d70440700 1 --2- 172.21.15.123:0/3420181641 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5d5402ff60 0x7f5d54032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f5d640096a0 tx=0x7f5d64006b40).stop 2024-01-13T11:51:16.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:16.167+0000 7f5d70440700 1 -- 172.21.15.123:0/3420181641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d680f5410 msgr2=0x7f5d68105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:16.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:16.167+0000 7f5d70440700 1 --2- 172.21.15.123:0/3420181641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d680f5410 0x7f5d68105630 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f5d58013510 tx=0x7f5d5800b040).stop 2024-01-13T11:51:16.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:16.167+0000 7f5d70440700 1 -- 172.21.15.123:0/3420181641 shutdown_connections 2024-01-13T11:51:16.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:16.167+0000 7f5d70440700 1 --2- 172.21.15.123:0/3420181641 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5d5402ff60 0x7f5d54032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:16.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:16.167+0000 7f5d70440700 1 --2- 172.21.15.123:0/3420181641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d680f5410 0x7f5d68105630 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:16.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:16.167+0000 7f5d70440700 1 -- 172.21.15.123:0/3420181641 >> 172.21.15.123:0/3420181641 conn(0x7f5d680f0b70 msgr2=0x7f5d680f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:16.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:16.167+0000 7f5d70440700 1 -- 172.21.15.123:0/3420181641 shutdown_connections 2024-01-13T11:51:16.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:16.167+0000 7f5d70440700 1 -- 172.21.15.123:0/3420181641 wait complete. 2024-01-13T11:51:16.157 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:16.888 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:16 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:15.514611+0000 mgr.smithi059.hmyldj (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:16.888 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:16 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:16.166542+0000 mon.smithi059 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.123:0/3420181641' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:17.796 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:17.797 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:18.888 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:18 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:17.514912+0000 mgr.smithi059.hmyldj (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:18.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.972+0000 7f560e692700 1 -- 172.21.15.123:0/2957279649 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5608060220 msgr2=0x7f56080f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:18.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.972+0000 7f560e692700 1 --2- 172.21.15.123:0/2957279649 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5608060220 0x7f56080f9150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f55fc009fb0 tx=0x7f55fc009b30).stop 2024-01-13T11:51:18.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.972+0000 7f560e692700 1 -- 172.21.15.123:0/2957279649 shutdown_connections 2024-01-13T11:51:18.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.972+0000 7f560e692700 1 --2- 172.21.15.123:0/2957279649 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5608060220 0x7f56080f9150 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:18.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.972+0000 7f560e692700 1 -- 172.21.15.123:0/2957279649 >> 172.21.15.123:0/2957279649 conn(0x7f56080f0b70 msgr2=0x7f56080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:18.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.972+0000 7f560e692700 1 -- 172.21.15.123:0/2957279649 shutdown_connections 2024-01-13T11:51:18.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.972+0000 7f560e692700 1 -- 172.21.15.123:0/2957279649 wait complete. 2024-01-13T11:51:18.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.972+0000 7f560e692700 1 Processor -- start 2024-01-13T11:51:18.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.973+0000 7f560e692700 1 -- start start 2024-01-13T11:51:18.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.973+0000 7f560e692700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5608060220 0x7f56081066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:18.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.973+0000 7f560e692700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5608106bd0 con 0x7f5608060220 2024-01-13T11:51:18.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.973+0000 7f5607fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5608060220 0x7f56081066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:18.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.973+0000 7f5607fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5608060220 0x7f56081066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:35344/0 (socket says 172.21.15.123:35344) 2024-01-13T11:51:18.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.973+0000 7f5607fff700 1 -- 172.21.15.123:0/1008947602 learned_addr learned my addr 172.21.15.123:0/1008947602 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:18.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.973+0000 7f5607fff700 1 -- 172.21.15.123:0/1008947602 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc005040 con 0x7f5608060220 2024-01-13T11:51:18.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.973+0000 7f5607fff700 1 --2- 172.21.15.123:0/1008947602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5608060220 0x7f56081066e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f55fc013510 tx=0x7f55fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:18.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.974+0000 7f5604ff9700 1 -- 172.21.15.123:0/1008947602 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55fc009320 con 0x7f5608060220 2024-01-13T11:51:18.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.974+0000 7f5604ff9700 1 -- 172.21.15.123:0/1008947602 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55fc013f00 con 0x7f5608060220 2024-01-13T11:51:18.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.974+0000 7f560e692700 1 -- 172.21.15.123:0/1008947602 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5608106dd0 con 0x7f5608060220 2024-01-13T11:51:18.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.974+0000 7f5604ff9700 1 -- 172.21.15.123:0/1008947602 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55fc014dc0 con 0x7f5608060220 2024-01-13T11:51:18.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.974+0000 7f560e692700 1 -- 172.21.15.123:0/1008947602 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5608107260 con 0x7f5608060220 2024-01-13T11:51:18.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.975+0000 7f5604ff9700 1 -- 172.21.15.123:0/1008947602 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f55fc0095f0 con 0x7f5608060220 2024-01-13T11:51:18.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.975+0000 7f560e692700 1 -- 172.21.15.123:0/1008947602 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5608045ad0 con 0x7f5608060220 2024-01-13T11:51:18.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.975+0000 7f5604ff9700 1 --2- 172.21.15.123:0/1008947602 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f55f002ff60 0x7f55f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:18.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.975+0000 7f5604ff9700 1 -- 172.21.15.123:0/1008947602 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f55fc041000 con 0x7f5608060220 2024-01-13T11:51:18.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.975+0000 7f56077fe700 1 --2- 172.21.15.123:0/1008947602 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f55f002ff60 0x7f55f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:18.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.975+0000 7f56077fe700 1 --2- 172.21.15.123:0/1008947602 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f55f002ff60 0x7f55f0032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f55f80096a0 tx=0x7f55f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:18.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:18.979+0000 7f5604ff9700 1 -- 172.21.15.123:0/1008947602 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f55fc01c790 con 0x7f5608060220 2024-01-13T11:51:19.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:19.187+0000 7f560e692700 1 -- 172.21.15.123:0/1008947602 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5608108110 con 0x7f5608060220 2024-01-13T11:51:19.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:19.188+0000 7f5604ff9700 1 -- 172.21.15.123:0/1008947602 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f55fc010230 con 0x7f5608060220 2024-01-13T11:51:19.177 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:19.177 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:19.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:19.189+0000 7f560e692700 1 -- 172.21.15.123:0/1008947602 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f55f002ff60 msgr2=0x7f55f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:19.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:19.189+0000 7f560e692700 1 --2- 172.21.15.123:0/1008947602 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f55f002ff60 0x7f55f0032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f55f80096a0 tx=0x7f55f8006b40).stop 2024-01-13T11:51:19.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:19.189+0000 7f560e692700 1 -- 172.21.15.123:0/1008947602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5608060220 msgr2=0x7f56081066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:19.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:19.189+0000 7f560e692700 1 --2- 172.21.15.123:0/1008947602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5608060220 0x7f56081066e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f55fc013510 tx=0x7f55fc00b040).stop 2024-01-13T11:51:19.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:19.189+0000 7f560e692700 1 -- 172.21.15.123:0/1008947602 shutdown_connections 2024-01-13T11:51:19.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:19.189+0000 7f560e692700 1 --2- 172.21.15.123:0/1008947602 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f55f002ff60 0x7f55f0032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:19.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:19.189+0000 7f560e692700 1 --2- 172.21.15.123:0/1008947602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5608060220 0x7f56081066e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:19.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:19.189+0000 7f560e692700 1 -- 172.21.15.123:0/1008947602 >> 172.21.15.123:0/1008947602 conn(0x7f56080f0b70 msgr2=0x7f56080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:19.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:19.189+0000 7f560e692700 1 -- 172.21.15.123:0/1008947602 shutdown_connections 2024-01-13T11:51:19.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:19.189+0000 7f560e692700 1 -- 172.21.15.123:0/1008947602 wait complete. 2024-01-13T11:51:19.180 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:19.888 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:19 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:19.189053+0000 mon.smithi059 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.123:0/1008947602' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:20.573 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:20.573 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:20.888 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:20 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:19.515240+0000 mgr.smithi059.hmyldj (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:21.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.715+0000 7fca2b039700 1 -- 172.21.15.123:0/1023641409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca240f6510 msgr2=0x7fca240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:21.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.715+0000 7fca2b039700 1 --2- 172.21.15.123:0/1023641409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca240f6510 0x7fca240f6930 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fca18009fb0 tx=0x7fca18009b30).stop 2024-01-13T11:51:21.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.715+0000 7fca2b039700 1 -- 172.21.15.123:0/1023641409 shutdown_connections 2024-01-13T11:51:21.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.715+0000 7fca2b039700 1 --2- 172.21.15.123:0/1023641409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca240f6510 0x7fca240f6930 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:21.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.715+0000 7fca2b039700 1 -- 172.21.15.123:0/1023641409 >> 172.21.15.123:0/1023641409 conn(0x7fca240f1c70 msgr2=0x7fca240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:21.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.715+0000 7fca2b039700 1 -- 172.21.15.123:0/1023641409 shutdown_connections 2024-01-13T11:51:21.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.715+0000 7fca2b039700 1 -- 172.21.15.123:0/1023641409 wait complete. 2024-01-13T11:51:21.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.716+0000 7fca2b039700 1 Processor -- start 2024-01-13T11:51:21.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.716+0000 7fca2b039700 1 -- start start 2024-01-13T11:51:21.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.716+0000 7fca2b039700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca240f6510 0x7fca24106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:21.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.716+0000 7fca2b039700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca24106c20 con 0x7fca240f6510 2024-01-13T11:51:21.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.717+0000 7fca28dd5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca240f6510 0x7fca24106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:21.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.717+0000 7fca28dd5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca240f6510 0x7fca24106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:35362/0 (socket says 172.21.15.123:35362) 2024-01-13T11:51:21.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.717+0000 7fca28dd5700 1 -- 172.21.15.123:0/1845064030 learned_addr learned my addr 172.21.15.123:0/1845064030 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:21.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.717+0000 7fca28dd5700 1 -- 172.21.15.123:0/1845064030 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca18005040 con 0x7fca240f6510 2024-01-13T11:51:21.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.717+0000 7fca28dd5700 1 --2- 172.21.15.123:0/1845064030 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca240f6510 0x7fca24106730 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fca18013510 tx=0x7fca1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:21.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.717+0000 7fca217fa700 1 -- 172.21.15.123:0/1845064030 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca18009320 con 0x7fca240f6510 2024-01-13T11:51:21.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.717+0000 7fca217fa700 1 -- 172.21.15.123:0/1845064030 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca18013f00 con 0x7fca240f6510 2024-01-13T11:51:21.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.717+0000 7fca2b039700 1 -- 172.21.15.123:0/1845064030 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca24106e20 con 0x7fca240f6510 2024-01-13T11:51:21.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.717+0000 7fca217fa700 1 -- 172.21.15.123:0/1845064030 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca18014d90 con 0x7fca240f6510 2024-01-13T11:51:21.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.717+0000 7fca2b039700 1 -- 172.21.15.123:0/1845064030 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca241072b0 con 0x7fca240f6510 2024-01-13T11:51:21.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.718+0000 7fca217fa700 1 -- 172.21.15.123:0/1845064030 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca180263a0 con 0x7fca240f6510 2024-01-13T11:51:21.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.718+0000 7fca2b039700 1 -- 172.21.15.123:0/1845064030 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca24045ad0 con 0x7fca240f6510 2024-01-13T11:51:21.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.718+0000 7fca217fa700 1 --2- 172.21.15.123:0/1845064030 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fca1402ff60 0x7fca14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:21.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.718+0000 7fca217fa700 1 -- 172.21.15.123:0/1845064030 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca18041010 con 0x7fca240f6510 2024-01-13T11:51:21.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.719+0000 7fca23fff700 1 --2- 172.21.15.123:0/1845064030 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fca1402ff60 0x7fca14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:21.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.719+0000 7fca23fff700 1 --2- 172.21.15.123:0/1845064030 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fca1402ff60 0x7fca14032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fca100096a0 tx=0x7fca10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:21.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.723+0000 7fca217fa700 1 -- 172.21.15.123:0/1845064030 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca180126f0 con 0x7fca240f6510 2024-01-13T11:51:21.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.929+0000 7fca2b039700 1 -- 172.21.15.123:0/1845064030 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca240fbf60 con 0x7fca240f6510 2024-01-13T11:51:21.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.930+0000 7fca217fa700 1 -- 172.21.15.123:0/1845064030 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fca18012330 con 0x7fca240f6510 2024-01-13T11:51:21.919 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:21.919 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:21.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.931+0000 7fca2b039700 1 -- 172.21.15.123:0/1845064030 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fca1402ff60 msgr2=0x7fca14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:21.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.931+0000 7fca2b039700 1 --2- 172.21.15.123:0/1845064030 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fca1402ff60 0x7fca14032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fca100096a0 tx=0x7fca10006b40).stop 2024-01-13T11:51:21.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.931+0000 7fca2b039700 1 -- 172.21.15.123:0/1845064030 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca240f6510 msgr2=0x7fca24106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:21.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.931+0000 7fca2b039700 1 --2- 172.21.15.123:0/1845064030 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca240f6510 0x7fca24106730 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fca18013510 tx=0x7fca1800b040).stop 2024-01-13T11:51:21.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.931+0000 7fca2b039700 1 -- 172.21.15.123:0/1845064030 shutdown_connections 2024-01-13T11:51:21.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.931+0000 7fca2b039700 1 --2- 172.21.15.123:0/1845064030 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fca1402ff60 0x7fca14032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:21.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.931+0000 7fca2b039700 1 --2- 172.21.15.123:0/1845064030 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca240f6510 0x7fca24106730 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:21.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.931+0000 7fca2b039700 1 -- 172.21.15.123:0/1845064030 >> 172.21.15.123:0/1845064030 conn(0x7fca240f1c70 msgr2=0x7fca240f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:21.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.931+0000 7fca2b039700 1 -- 172.21.15.123:0/1845064030 shutdown_connections 2024-01-13T11:51:21.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:21.931+0000 7fca2b039700 1 -- 172.21.15.123:0/1845064030 wait complete. 2024-01-13T11:51:21.921 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:22.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:22 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:21.930843+0000 mon.smithi059 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.123:0/1845064030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:23.318 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:23.318 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:23.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:23 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:21.515531+0000 mgr.smithi059.hmyldj (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:24.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.462+0000 7fe881f2c700 1 -- 172.21.15.123:0/983482992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe87c0f6510 msgr2=0x7fe87c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:24.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.462+0000 7fe881f2c700 1 --2- 172.21.15.123:0/983482992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe87c0f6510 0x7fe87c0f6930 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fe870009fb0 tx=0x7fe870009b30).stop 2024-01-13T11:51:24.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.462+0000 7fe881f2c700 1 -- 172.21.15.123:0/983482992 shutdown_connections 2024-01-13T11:51:24.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.462+0000 7fe881f2c700 1 --2- 172.21.15.123:0/983482992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe87c0f6510 0x7fe87c0f6930 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:24.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.462+0000 7fe881f2c700 1 -- 172.21.15.123:0/983482992 >> 172.21.15.123:0/983482992 conn(0x7fe87c0f1c70 msgr2=0x7fe87c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:24.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.462+0000 7fe881f2c700 1 -- 172.21.15.123:0/983482992 shutdown_connections 2024-01-13T11:51:24.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.462+0000 7fe881f2c700 1 -- 172.21.15.123:0/983482992 wait complete. 2024-01-13T11:51:24.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.463+0000 7fe881f2c700 1 Processor -- start 2024-01-13T11:51:24.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.463+0000 7fe881f2c700 1 -- start start 2024-01-13T11:51:24.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.463+0000 7fe881f2c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe87c0f6510 0x7fe87c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:24.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.463+0000 7fe881f2c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c106c20 con 0x7fe87c0f6510 2024-01-13T11:51:24.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.463+0000 7fe87b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe87c0f6510 0x7fe87c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:24.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.463+0000 7fe87b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe87c0f6510 0x7fe87c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:35382/0 (socket says 172.21.15.123:35382) 2024-01-13T11:51:24.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.463+0000 7fe87b7fe700 1 -- 172.21.15.123:0/3719091085 learned_addr learned my addr 172.21.15.123:0/3719091085 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:24.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.463+0000 7fe87b7fe700 1 -- 172.21.15.123:0/3719091085 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe870005040 con 0x7fe87c0f6510 2024-01-13T11:51:24.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.464+0000 7fe87b7fe700 1 --2- 172.21.15.123:0/3719091085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe87c0f6510 0x7fe87c106730 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fe870013510 tx=0x7fe87000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:24.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.464+0000 7fe863fff700 1 -- 172.21.15.123:0/3719091085 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe870009320 con 0x7fe87c0f6510 2024-01-13T11:51:24.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.464+0000 7fe881f2c700 1 -- 172.21.15.123:0/3719091085 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe87c106e20 con 0x7fe87c0f6510 2024-01-13T11:51:24.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.464+0000 7fe863fff700 1 -- 172.21.15.123:0/3719091085 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe870013f00 con 0x7fe87c0f6510 2024-01-13T11:51:24.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.464+0000 7fe863fff700 1 -- 172.21.15.123:0/3719091085 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe870014d90 con 0x7fe87c0f6510 2024-01-13T11:51:24.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.464+0000 7fe881f2c700 1 -- 172.21.15.123:0/3719091085 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe87c1072b0 con 0x7fe87c0f6510 2024-01-13T11:51:24.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.465+0000 7fe863fff700 1 -- 172.21.15.123:0/3719091085 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8700095f0 con 0x7fe87c0f6510 2024-01-13T11:51:24.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.465+0000 7fe881f2c700 1 -- 172.21.15.123:0/3719091085 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe87c045ad0 con 0x7fe87c0f6510 2024-01-13T11:51:24.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.465+0000 7fe863fff700 1 --2- 172.21.15.123:0/3719091085 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe86402ff60 0x7fe864032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:24.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.465+0000 7fe863fff700 1 -- 172.21.15.123:0/3719091085 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe8700410c0 con 0x7fe87c0f6510 2024-01-13T11:51:24.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.465+0000 7fe87affd700 1 --2- 172.21.15.123:0/3719091085 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe86402ff60 0x7fe864032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:24.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.465+0000 7fe87affd700 1 --2- 172.21.15.123:0/3719091085 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe86402ff60 0x7fe864032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fe86c0096a0 tx=0x7fe86c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:24.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.469+0000 7fe863fff700 1 -- 172.21.15.123:0/3719091085 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe870014420 con 0x7fe87c0f6510 2024-01-13T11:51:24.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.676+0000 7fe881f2c700 1 -- 172.21.15.123:0/3719091085 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe87c0fbf60 con 0x7fe87c0f6510 2024-01-13T11:51:24.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.677+0000 7fe863fff700 1 -- 172.21.15.123:0/3719091085 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe87001d7f0 con 0x7fe87c0f6510 2024-01-13T11:51:24.667 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:24.667 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:24.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.679+0000 7fe881f2c700 1 -- 172.21.15.123:0/3719091085 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe86402ff60 msgr2=0x7fe864032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:24.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.679+0000 7fe881f2c700 1 --2- 172.21.15.123:0/3719091085 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe86402ff60 0x7fe864032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fe86c0096a0 tx=0x7fe86c006b40).stop 2024-01-13T11:51:24.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.679+0000 7fe881f2c700 1 -- 172.21.15.123:0/3719091085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe87c0f6510 msgr2=0x7fe87c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:24.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.679+0000 7fe881f2c700 1 --2- 172.21.15.123:0/3719091085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe87c0f6510 0x7fe87c106730 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fe870013510 tx=0x7fe87000b040).stop 2024-01-13T11:51:24.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.679+0000 7fe881f2c700 1 -- 172.21.15.123:0/3719091085 shutdown_connections 2024-01-13T11:51:24.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.679+0000 7fe881f2c700 1 --2- 172.21.15.123:0/3719091085 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe86402ff60 0x7fe864032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:24.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.679+0000 7fe881f2c700 1 --2- 172.21.15.123:0/3719091085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe87c0f6510 0x7fe87c106730 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:24.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.679+0000 7fe881f2c700 1 -- 172.21.15.123:0/3719091085 >> 172.21.15.123:0/3719091085 conn(0x7fe87c0f1c70 msgr2=0x7fe87c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:24.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.679+0000 7fe881f2c700 1 -- 172.21.15.123:0/3719091085 shutdown_connections 2024-01-13T11:51:24.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:24.679+0000 7fe881f2c700 1 -- 172.21.15.123:0/3719091085 wait complete. 2024-01-13T11:51:24.669 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:25.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:25 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:23.515819+0000 mgr.smithi059.hmyldj (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:25.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:25 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:24.678451+0000 mon.smithi059 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.123:0/3719091085' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:26.102 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:26.102 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:27.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.236+0000 7f83cb276700 1 -- 172.21.15.123:0/2154775377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83c40f5410 msgr2=0x7f83c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:27.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.236+0000 7f83cb276700 1 --2- 172.21.15.123:0/2154775377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83c40f5410 0x7f83c40f5830 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f83b8009fb0 tx=0x7f83b8009b30).stop 2024-01-13T11:51:27.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.236+0000 7f83cb276700 1 -- 172.21.15.123:0/2154775377 shutdown_connections 2024-01-13T11:51:27.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.236+0000 7f83cb276700 1 --2- 172.21.15.123:0/2154775377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83c40f5410 0x7f83c40f5830 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:27.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.236+0000 7f83cb276700 1 -- 172.21.15.123:0/2154775377 >> 172.21.15.123:0/2154775377 conn(0x7f83c40f0b70 msgr2=0x7f83c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:27.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.236+0000 7f83cb276700 1 -- 172.21.15.123:0/2154775377 shutdown_connections 2024-01-13T11:51:27.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.236+0000 7f83cb276700 1 -- 172.21.15.123:0/2154775377 wait complete. 2024-01-13T11:51:27.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.237+0000 7f83cb276700 1 Processor -- start 2024-01-13T11:51:27.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.237+0000 7f83cb276700 1 -- start start 2024-01-13T11:51:27.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.237+0000 7f83cb276700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83c40f5410 0x7f83c4108970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:27.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.237+0000 7f83cb276700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c4108e60 con 0x7f83c40f5410 2024-01-13T11:51:27.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.237+0000 7f83c9012700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83c40f5410 0x7f83c4108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:27.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.237+0000 7f83c9012700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83c40f5410 0x7f83c4108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:60242/0 (socket says 172.21.15.123:60242) 2024-01-13T11:51:27.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.237+0000 7f83c9012700 1 -- 172.21.15.123:0/3092281907 learned_addr learned my addr 172.21.15.123:0/3092281907 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:27.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.237+0000 7f83c9012700 1 -- 172.21.15.123:0/3092281907 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b8005040 con 0x7f83c40f5410 2024-01-13T11:51:27.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.237+0000 7f83c9012700 1 --2- 172.21.15.123:0/3092281907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83c40f5410 0x7f83c4108970 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f83b8009f80 tx=0x7f83b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:27.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.238+0000 7f83b5ffb700 1 -- 172.21.15.123:0/3092281907 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83b8009320 con 0x7f83c40f5410 2024-01-13T11:51:27.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.238+0000 7f83cb276700 1 -- 172.21.15.123:0/3092281907 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c4109060 con 0x7f83c40f5410 2024-01-13T11:51:27.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.238+0000 7f83b5ffb700 1 -- 172.21.15.123:0/3092281907 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f83b8013ec0 con 0x7f83c40f5410 2024-01-13T11:51:27.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.238+0000 7f83b5ffb700 1 -- 172.21.15.123:0/3092281907 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83b801ece0 con 0x7f83c40f5410 2024-01-13T11:51:27.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.238+0000 7f83cb276700 1 -- 172.21.15.123:0/3092281907 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c41094f0 con 0x7f83c40f5410 2024-01-13T11:51:27.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.239+0000 7f83b5ffb700 1 -- 172.21.15.123:0/3092281907 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f83b801e420 con 0x7f83c40f5410 2024-01-13T11:51:27.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.239+0000 7f83cb276700 1 -- 172.21.15.123:0/3092281907 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c4045ad0 con 0x7f83c40f5410 2024-01-13T11:51:27.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.239+0000 7f83b5ffb700 1 --2- 172.21.15.123:0/3092281907 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f83b002ff60 0x7f83b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:27.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.239+0000 7f83b5ffb700 1 -- 172.21.15.123:0/3092281907 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f83b8040db0 con 0x7f83c40f5410 2024-01-13T11:51:27.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.239+0000 7f83c8811700 1 --2- 172.21.15.123:0/3092281907 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f83b002ff60 0x7f83b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:27.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.240+0000 7f83c8811700 1 --2- 172.21.15.123:0/3092281907 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f83b002ff60 0x7f83b0032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f83c00096a0 tx=0x7f83c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:27.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.244+0000 7f83b5ffb700 1 -- 172.21.15.123:0/3092281907 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f83b801e730 con 0x7f83c40f5410 2024-01-13T11:51:27.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.450+0000 7f83cb276700 1 -- 172.21.15.123:0/3092281907 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83c410a3a0 con 0x7f83c40f5410 2024-01-13T11:51:27.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.451+0000 7f83b5ffb700 1 -- 172.21.15.123:0/3092281907 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f83b8004490 con 0x7f83c40f5410 2024-01-13T11:51:27.440 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:27.440 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:27.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.452+0000 7f83cb276700 1 -- 172.21.15.123:0/3092281907 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f83b002ff60 msgr2=0x7f83b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:27.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.452+0000 7f83cb276700 1 --2- 172.21.15.123:0/3092281907 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f83b002ff60 0x7f83b0032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f83c00096a0 tx=0x7f83c0006b40).stop 2024-01-13T11:51:27.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.452+0000 7f83cb276700 1 -- 172.21.15.123:0/3092281907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83c40f5410 msgr2=0x7f83c4108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:27.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.452+0000 7f83cb276700 1 --2- 172.21.15.123:0/3092281907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83c40f5410 0x7f83c4108970 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f83b8009f80 tx=0x7f83b800b040).stop 2024-01-13T11:51:27.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.452+0000 7f83cb276700 1 -- 172.21.15.123:0/3092281907 shutdown_connections 2024-01-13T11:51:27.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.452+0000 7f83cb276700 1 --2- 172.21.15.123:0/3092281907 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f83b002ff60 0x7f83b0032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:27.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.452+0000 7f83cb276700 1 --2- 172.21.15.123:0/3092281907 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83c40f5410 0x7f83c4108970 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:27.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.452+0000 7f83cb276700 1 -- 172.21.15.123:0/3092281907 >> 172.21.15.123:0/3092281907 conn(0x7f83c40f0b70 msgr2=0x7f83c405fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:27.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.452+0000 7f83cb276700 1 -- 172.21.15.123:0/3092281907 shutdown_connections 2024-01-13T11:51:27.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:27.453+0000 7f83cb276700 1 -- 172.21.15.123:0/3092281907 wait complete. 2024-01-13T11:51:27.443 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:27.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:27 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:25.516058+0000 mgr.smithi059.hmyldj (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:28.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:28 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:27.452177+0000 mon.smithi059 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.123:0/3092281907' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:28.835 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:28.835 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:29.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:29 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:27.516340+0000 mgr.smithi059.hmyldj (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:29.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.946+0000 7fb9c7543700 1 -- 172.21.15.123:0/2737912348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9c00f51a0 msgr2=0x7fb9c00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:29.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.946+0000 7fb9c7543700 1 --2- 172.21.15.123:0/2737912348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9c00f51a0 0x7fb9c00f55c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fb9bc009fb0 tx=0x7fb9bc009b30).stop 2024-01-13T11:51:29.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.946+0000 7fb9c7543700 1 -- 172.21.15.123:0/2737912348 shutdown_connections 2024-01-13T11:51:29.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.946+0000 7fb9c7543700 1 --2- 172.21.15.123:0/2737912348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9c00f51a0 0x7fb9c00f55c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:29.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.946+0000 7fb9c7543700 1 -- 172.21.15.123:0/2737912348 >> 172.21.15.123:0/2737912348 conn(0x7fb9c00f09a0 msgr2=0x7fb9c00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:29.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.946+0000 7fb9c7543700 1 -- 172.21.15.123:0/2737912348 shutdown_connections 2024-01-13T11:51:29.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.946+0000 7fb9c7543700 1 -- 172.21.15.123:0/2737912348 wait complete. 2024-01-13T11:51:29.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.947+0000 7fb9c7543700 1 Processor -- start 2024-01-13T11:51:29.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.947+0000 7fb9c7543700 1 -- start start 2024-01-13T11:51:29.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.947+0000 7fb9c7543700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9c00f51a0 0x7fb9c01053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:29.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.947+0000 7fb9c7543700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c01058a0 con 0x7fb9c00f51a0 2024-01-13T11:51:29.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.947+0000 7fb9c6541700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9c00f51a0 0x7fb9c01053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:29.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.947+0000 7fb9c6541700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9c00f51a0 0x7fb9c01053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:60262/0 (socket says 172.21.15.123:60262) 2024-01-13T11:51:29.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.947+0000 7fb9c6541700 1 -- 172.21.15.123:0/3843043814 learned_addr learned my addr 172.21.15.123:0/3843043814 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:29.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.947+0000 7fb9c6541700 1 -- 172.21.15.123:0/3843043814 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9bc005040 con 0x7fb9c00f51a0 2024-01-13T11:51:29.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.947+0000 7fb9c6541700 1 --2- 172.21.15.123:0/3843043814 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9c00f51a0 0x7fb9c01053b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fb9bc013510 tx=0x7fb9bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:29.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.948+0000 7fb9b6ffd700 1 -- 172.21.15.123:0/3843043814 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9bc009320 con 0x7fb9c00f51a0 2024-01-13T11:51:29.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.948+0000 7fb9c7543700 1 -- 172.21.15.123:0/3843043814 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c0105aa0 con 0x7fb9c00f51a0 2024-01-13T11:51:29.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.948+0000 7fb9b6ffd700 1 -- 172.21.15.123:0/3843043814 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9bc013f00 con 0x7fb9c00f51a0 2024-01-13T11:51:29.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.948+0000 7fb9b6ffd700 1 -- 172.21.15.123:0/3843043814 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9bc014dc0 con 0x7fb9c00f51a0 2024-01-13T11:51:29.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.948+0000 7fb9c7543700 1 -- 172.21.15.123:0/3843043814 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c0105f30 con 0x7fb9c00f51a0 2024-01-13T11:51:29.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.948+0000 7fb9b6ffd700 1 -- 172.21.15.123:0/3843043814 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9bc0095f0 con 0x7fb9c00f51a0 2024-01-13T11:51:29.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.948+0000 7fb9c7543700 1 -- 172.21.15.123:0/3843043814 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c0045ad0 con 0x7fb9c00f51a0 2024-01-13T11:51:29.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.949+0000 7fb9b6ffd700 1 --2- 172.21.15.123:0/3843043814 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb9ac02ff60 0x7fb9ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:29.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.949+0000 7fb9b6ffd700 1 -- 172.21.15.123:0/3843043814 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb9bc041030 con 0x7fb9c00f51a0 2024-01-13T11:51:29.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.949+0000 7fb9c5d40700 1 --2- 172.21.15.123:0/3843043814 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb9ac02ff60 0x7fb9ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:29.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.949+0000 7fb9c5d40700 1 --2- 172.21.15.123:0/3843043814 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb9ac02ff60 0x7fb9ac032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fb9b00096a0 tx=0x7fb9b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:29.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:29.953+0000 7fb9b6ffd700 1 -- 172.21.15.123:0/3843043814 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9bc014420 con 0x7fb9c00f51a0 2024-01-13T11:51:30.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:30.160+0000 7fb9c7543700 1 -- 172.21.15.123:0/3843043814 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9c0106de0 con 0x7fb9c00f51a0 2024-01-13T11:51:30.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:30.161+0000 7fb9b6ffd700 1 -- 172.21.15.123:0/3843043814 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb9bc010230 con 0x7fb9c00f51a0 2024-01-13T11:51:30.150 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:30.150 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:30.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:30.162+0000 7fb9c7543700 1 -- 172.21.15.123:0/3843043814 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb9ac02ff60 msgr2=0x7fb9ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:30.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:30.162+0000 7fb9c7543700 1 --2- 172.21.15.123:0/3843043814 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb9ac02ff60 0x7fb9ac032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fb9b00096a0 tx=0x7fb9b0006b40).stop 2024-01-13T11:51:30.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:30.162+0000 7fb9c7543700 1 -- 172.21.15.123:0/3843043814 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9c00f51a0 msgr2=0x7fb9c01053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:30.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:30.162+0000 7fb9c7543700 1 --2- 172.21.15.123:0/3843043814 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9c00f51a0 0x7fb9c01053b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fb9bc013510 tx=0x7fb9bc00b040).stop 2024-01-13T11:51:30.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:30.162+0000 7fb9c7543700 1 -- 172.21.15.123:0/3843043814 shutdown_connections 2024-01-13T11:51:30.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:30.162+0000 7fb9c7543700 1 --2- 172.21.15.123:0/3843043814 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb9ac02ff60 0x7fb9ac032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:30.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:30.162+0000 7fb9c7543700 1 --2- 172.21.15.123:0/3843043814 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9c00f51a0 0x7fb9c01053b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:30.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:30.162+0000 7fb9c7543700 1 -- 172.21.15.123:0/3843043814 >> 172.21.15.123:0/3843043814 conn(0x7fb9c00f09a0 msgr2=0x7fb9c00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:30.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:30.162+0000 7fb9c7543700 1 -- 172.21.15.123:0/3843043814 shutdown_connections 2024-01-13T11:51:30.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:30.163+0000 7fb9c7543700 1 -- 172.21.15.123:0/3843043814 wait complete. 2024-01-13T11:51:30.153 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:30.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:30 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:30.161960+0000 mon.smithi059 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.123:0/3843043814' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:31.533 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:31.533 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:31.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:31 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:29.516657+0000 mgr.smithi059.hmyldj (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:32.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.696+0000 7ffbb1de4700 1 -- 172.21.15.123:0/1881287601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbac0f5410 msgr2=0x7ffbac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:32.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.696+0000 7ffbb1de4700 1 --2- 172.21.15.123:0/1881287601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbac0f5410 0x7ffbac0f5830 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7ffb9c009fb0 tx=0x7ffb9c009b30).stop 2024-01-13T11:51:32.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.696+0000 7ffbb1de4700 1 -- 172.21.15.123:0/1881287601 shutdown_connections 2024-01-13T11:51:32.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.696+0000 7ffbb1de4700 1 --2- 172.21.15.123:0/1881287601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbac0f5410 0x7ffbac0f5830 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:32.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.696+0000 7ffbb1de4700 1 -- 172.21.15.123:0/1881287601 >> 172.21.15.123:0/1881287601 conn(0x7ffbac0f0b70 msgr2=0x7ffbac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:32.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.697+0000 7ffbb1de4700 1 -- 172.21.15.123:0/1881287601 shutdown_connections 2024-01-13T11:51:32.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.697+0000 7ffbb1de4700 1 -- 172.21.15.123:0/1881287601 wait complete. 2024-01-13T11:51:32.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.697+0000 7ffbb1de4700 1 Processor -- start 2024-01-13T11:51:32.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.697+0000 7ffbb1de4700 1 -- start start 2024-01-13T11:51:32.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.697+0000 7ffbb1de4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbac0f5410 0x7ffbac109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:32.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.697+0000 7ffbb1de4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac060240 con 0x7ffbac0f5410 2024-01-13T11:51:32.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.698+0000 7ffbab7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbac0f5410 0x7ffbac109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:32.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.698+0000 7ffbab7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbac0f5410 0x7ffbac109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:60266/0 (socket says 172.21.15.123:60266) 2024-01-13T11:51:32.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.698+0000 7ffbab7fe700 1 -- 172.21.15.123:0/782189939 learned_addr learned my addr 172.21.15.123:0/782189939 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:32.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.698+0000 7ffbab7fe700 1 -- 172.21.15.123:0/782189939 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9c005040 con 0x7ffbac0f5410 2024-01-13T11:51:32.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.698+0000 7ffbab7fe700 1 --2- 172.21.15.123:0/782189939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbac0f5410 0x7ffbac109430 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7ffb9c013510 tx=0x7ffb9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:32.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.698+0000 7ffb93fff700 1 -- 172.21.15.123:0/782189939 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb9c009320 con 0x7ffbac0f5410 2024-01-13T11:51:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.698+0000 7ffbb1de4700 1 -- 172.21.15.123:0/782189939 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbac060440 con 0x7ffbac0f5410 2024-01-13T11:51:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.698+0000 7ffb93fff700 1 -- 172.21.15.123:0/782189939 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb9c013f00 con 0x7ffbac0f5410 2024-01-13T11:51:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.698+0000 7ffb93fff700 1 -- 172.21.15.123:0/782189939 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb9c014d90 con 0x7ffbac0f5410 2024-01-13T11:51:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.698+0000 7ffbb1de4700 1 -- 172.21.15.123:0/782189939 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbac109ae0 con 0x7ffbac0f5410 2024-01-13T11:51:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.699+0000 7ffb93fff700 1 -- 172.21.15.123:0/782189939 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffb9c0095f0 con 0x7ffbac0f5410 2024-01-13T11:51:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.699+0000 7ffbb1de4700 1 -- 172.21.15.123:0/782189939 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbac045ad0 con 0x7ffbac0f5410 2024-01-13T11:51:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.699+0000 7ffb93fff700 1 --2- 172.21.15.123:0/782189939 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffb9402ff60 0x7ffb94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.699+0000 7ffb93fff700 1 -- 172.21.15.123:0/782189939 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffb9c0410c0 con 0x7ffbac0f5410 2024-01-13T11:51:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.699+0000 7ffbaaffd700 1 --2- 172.21.15.123:0/782189939 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffb9402ff60 0x7ffb94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:32.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.700+0000 7ffbaaffd700 1 --2- 172.21.15.123:0/782189939 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffb9402ff60 0x7ffb94032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7ffba00096a0 tx=0x7ffba0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:32.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.704+0000 7ffb93fff700 1 -- 172.21.15.123:0/782189939 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb9c014420 con 0x7ffbac0f5410 2024-01-13T11:51:32.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.912+0000 7ffbb1de4700 1 -- 172.21.15.123:0/782189939 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffbac0fae60 con 0x7ffbac0f5410 2024-01-13T11:51:32.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.913+0000 7ffb93fff700 1 -- 172.21.15.123:0/782189939 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffb9c010230 con 0x7ffbac0f5410 2024-01-13T11:51:32.902 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:32.902 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:32.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.914+0000 7ffbb1de4700 1 -- 172.21.15.123:0/782189939 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffb9402ff60 msgr2=0x7ffb94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:32.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.914+0000 7ffbb1de4700 1 --2- 172.21.15.123:0/782189939 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffb9402ff60 0x7ffb94032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7ffba00096a0 tx=0x7ffba0006b40).stop 2024-01-13T11:51:32.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.914+0000 7ffbb1de4700 1 -- 172.21.15.123:0/782189939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbac0f5410 msgr2=0x7ffbac109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:32.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.914+0000 7ffbb1de4700 1 --2- 172.21.15.123:0/782189939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbac0f5410 0x7ffbac109430 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7ffb9c013510 tx=0x7ffb9c00b040).stop 2024-01-13T11:51:32.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.914+0000 7ffbb1de4700 1 -- 172.21.15.123:0/782189939 shutdown_connections 2024-01-13T11:51:32.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.914+0000 7ffbb1de4700 1 --2- 172.21.15.123:0/782189939 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffb9402ff60 0x7ffb94032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:32.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.914+0000 7ffbb1de4700 1 --2- 172.21.15.123:0/782189939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbac0f5410 0x7ffbac109430 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:32.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.914+0000 7ffbb1de4700 1 -- 172.21.15.123:0/782189939 >> 172.21.15.123:0/782189939 conn(0x7ffbac0f0b70 msgr2=0x7ffbac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:32.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.915+0000 7ffbb1de4700 1 -- 172.21.15.123:0/782189939 shutdown_connections 2024-01-13T11:51:32.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:32.915+0000 7ffbb1de4700 1 -- 172.21.15.123:0/782189939 wait complete. 2024-01-13T11:51:32.905 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:33.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:33 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:31.516937+0000 mgr.smithi059.hmyldj (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:33.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:33 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:32.914153+0000 mon.smithi059 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.123:0/782189939' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:34.289 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:34.289 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:35.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.394+0000 7ff5c88c3700 1 -- 172.21.15.123:0/4143911078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5c40f51a0 msgr2=0x7ff5c40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:35.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.394+0000 7ff5c88c3700 1 --2- 172.21.15.123:0/4143911078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5c40f51a0 0x7ff5c40f55c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7ff5b8009fb0 tx=0x7ff5b8009b30).stop 2024-01-13T11:51:35.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.394+0000 7ff5c88c3700 1 -- 172.21.15.123:0/4143911078 shutdown_connections 2024-01-13T11:51:35.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.394+0000 7ff5c88c3700 1 --2- 172.21.15.123:0/4143911078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5c40f51a0 0x7ff5c40f55c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:35.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.394+0000 7ff5c88c3700 1 -- 172.21.15.123:0/4143911078 >> 172.21.15.123:0/4143911078 conn(0x7ff5c40f09a0 msgr2=0x7ff5c40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:35.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.394+0000 7ff5c88c3700 1 -- 172.21.15.123:0/4143911078 shutdown_connections 2024-01-13T11:51:35.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.394+0000 7ff5c88c3700 1 -- 172.21.15.123:0/4143911078 wait complete. 2024-01-13T11:51:35.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.395+0000 7ff5c88c3700 1 Processor -- start 2024-01-13T11:51:35.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.395+0000 7ff5c88c3700 1 -- start start 2024-01-13T11:51:35.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.395+0000 7ff5c88c3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5c40f51a0 0x7ff5c41064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:35.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.395+0000 7ff5c88c3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c4106990 con 0x7ff5c40f51a0 2024-01-13T11:51:35.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.395+0000 7ff5c2d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5c40f51a0 0x7ff5c41064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:35.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.395+0000 7ff5c2d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5c40f51a0 0x7ff5c41064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:60272/0 (socket says 172.21.15.123:60272) 2024-01-13T11:51:35.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.395+0000 7ff5c2d9d700 1 -- 172.21.15.123:0/2566060198 learned_addr learned my addr 172.21.15.123:0/2566060198 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:35.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.396+0000 7ff5c2d9d700 1 -- 172.21.15.123:0/2566060198 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b8005040 con 0x7ff5c40f51a0 2024-01-13T11:51:35.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.396+0000 7ff5c2d9d700 1 --2- 172.21.15.123:0/2566060198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5c40f51a0 0x7ff5c41064a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ff5b8013510 tx=0x7ff5b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:35.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.396+0000 7ff5ab7fe700 1 -- 172.21.15.123:0/2566060198 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5b8009320 con 0x7ff5c40f51a0 2024-01-13T11:51:35.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.396+0000 7ff5c88c3700 1 -- 172.21.15.123:0/2566060198 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c4106b90 con 0x7ff5c40f51a0 2024-01-13T11:51:35.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.396+0000 7ff5ab7fe700 1 -- 172.21.15.123:0/2566060198 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff5b8013f00 con 0x7ff5c40f51a0 2024-01-13T11:51:35.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.396+0000 7ff5ab7fe700 1 -- 172.21.15.123:0/2566060198 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5b8014dc0 con 0x7ff5c40f51a0 2024-01-13T11:51:35.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.396+0000 7ff5c88c3700 1 -- 172.21.15.123:0/2566060198 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c4107020 con 0x7ff5c40f51a0 2024-01-13T11:51:35.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.397+0000 7ff5c88c3700 1 -- 172.21.15.123:0/2566060198 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5c4045ad0 con 0x7ff5c40f51a0 2024-01-13T11:51:35.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.397+0000 7ff5ab7fe700 1 -- 172.21.15.123:0/2566060198 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff5b80095f0 con 0x7ff5c40f51a0 2024-01-13T11:51:35.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.397+0000 7ff5ab7fe700 1 --2- 172.21.15.123:0/2566060198 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff5ac02ff60 0x7ff5ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:35.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.397+0000 7ff5ab7fe700 1 -- 172.21.15.123:0/2566060198 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff5b8041020 con 0x7ff5c40f51a0 2024-01-13T11:51:35.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.397+0000 7ff5c259c700 1 --2- 172.21.15.123:0/2566060198 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff5ac02ff60 0x7ff5ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:35.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.398+0000 7ff5c259c700 1 --2- 172.21.15.123:0/2566060198 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff5ac02ff60 0x7ff5ac032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7ff5b40096a0 tx=0x7ff5b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:35.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.402+0000 7ff5ab7fe700 1 -- 172.21.15.123:0/2566060198 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff5b8014420 con 0x7ff5c40f51a0 2024-01-13T11:51:35.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.611+0000 7ff5c88c3700 1 -- 172.21.15.123:0/2566060198 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff5c4107e70 con 0x7ff5c40f51a0 2024-01-13T11:51:35.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.611+0000 7ff5ab7fe700 1 -- 172.21.15.123:0/2566060198 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff5b801c920 con 0x7ff5c40f51a0 2024-01-13T11:51:35.601 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:35.601 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:35.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.613+0000 7ff5c88c3700 1 -- 172.21.15.123:0/2566060198 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff5ac02ff60 msgr2=0x7ff5ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:35.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.613+0000 7ff5c88c3700 1 --2- 172.21.15.123:0/2566060198 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff5ac02ff60 0x7ff5ac032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7ff5b40096a0 tx=0x7ff5b4006b40).stop 2024-01-13T11:51:35.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.613+0000 7ff5c88c3700 1 -- 172.21.15.123:0/2566060198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5c40f51a0 msgr2=0x7ff5c41064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:35.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.613+0000 7ff5c88c3700 1 --2- 172.21.15.123:0/2566060198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5c40f51a0 0x7ff5c41064a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ff5b8013510 tx=0x7ff5b800b040).stop 2024-01-13T11:51:35.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.613+0000 7ff5c88c3700 1 -- 172.21.15.123:0/2566060198 shutdown_connections 2024-01-13T11:51:35.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.613+0000 7ff5c88c3700 1 --2- 172.21.15.123:0/2566060198 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff5ac02ff60 0x7ff5ac032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:35.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.613+0000 7ff5c88c3700 1 --2- 172.21.15.123:0/2566060198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5c40f51a0 0x7ff5c41064a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:35.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.613+0000 7ff5c88c3700 1 -- 172.21.15.123:0/2566060198 >> 172.21.15.123:0/2566060198 conn(0x7ff5c40f09a0 msgr2=0x7ff5c405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:35.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.613+0000 7ff5c88c3700 1 -- 172.21.15.123:0/2566060198 shutdown_connections 2024-01-13T11:51:35.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:35.613+0000 7ff5c88c3700 1 -- 172.21.15.123:0/2566060198 wait complete. 2024-01-13T11:51:35.604 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:35.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:35 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:33.517191+0000 mgr.smithi059.hmyldj (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:36.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:36 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:35.612687+0000 mon.smithi059 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.123:0/2566060198' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:36.992 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:36.993 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:37.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:37 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:35.517494+0000 mgr.smithi059.hmyldj (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:38.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.197+0000 7f37c117b700 1 -- 172.21.15.123:0/2718355749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37bc0f5410 msgr2=0x7f37bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:38.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.197+0000 7f37c117b700 1 --2- 172.21.15.123:0/2718355749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37bc0f5410 0x7f37bc0f5830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f37a4009fb0 tx=0x7f37a4009b30).stop 2024-01-13T11:51:38.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.197+0000 7f37c117b700 1 -- 172.21.15.123:0/2718355749 shutdown_connections 2024-01-13T11:51:38.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.197+0000 7f37c117b700 1 --2- 172.21.15.123:0/2718355749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37bc0f5410 0x7f37bc0f5830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:38.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.197+0000 7f37c117b700 1 -- 172.21.15.123:0/2718355749 >> 172.21.15.123:0/2718355749 conn(0x7f37bc0f0b70 msgr2=0x7f37bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:38.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.197+0000 7f37c117b700 1 -- 172.21.15.123:0/2718355749 shutdown_connections 2024-01-13T11:51:38.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.197+0000 7f37c117b700 1 -- 172.21.15.123:0/2718355749 wait complete. 2024-01-13T11:51:38.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.198+0000 7f37c117b700 1 Processor -- start 2024-01-13T11:51:38.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.198+0000 7f37c117b700 1 -- start start 2024-01-13T11:51:38.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.198+0000 7f37c117b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37bc0f5410 0x7f37bc109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:38.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.198+0000 7f37c117b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37bc060220 con 0x7f37bc0f5410 2024-01-13T11:51:38.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.198+0000 7f37bad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37bc0f5410 0x7f37bc109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:38.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.198+0000 7f37bad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37bc0f5410 0x7f37bc109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:54326/0 (socket says 172.21.15.123:54326) 2024-01-13T11:51:38.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.198+0000 7f37bad9d700 1 -- 172.21.15.123:0/1217943504 learned_addr learned my addr 172.21.15.123:0/1217943504 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:38.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.199+0000 7f37bad9d700 1 -- 172.21.15.123:0/1217943504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a4005040 con 0x7f37bc0f5410 2024-01-13T11:51:38.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.199+0000 7f37bad9d700 1 --2- 172.21.15.123:0/1217943504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37bc0f5410 0x7f37bc109450 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f37a4013510 tx=0x7f37a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:38.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.199+0000 7f37b37fe700 1 -- 172.21.15.123:0/1217943504 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37a4009320 con 0x7f37bc0f5410 2024-01-13T11:51:38.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.199+0000 7f37c117b700 1 -- 172.21.15.123:0/1217943504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37bc060420 con 0x7f37bc0f5410 2024-01-13T11:51:38.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.199+0000 7f37b37fe700 1 -- 172.21.15.123:0/1217943504 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37a4013f00 con 0x7f37bc0f5410 2024-01-13T11:51:38.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.199+0000 7f37b37fe700 1 -- 172.21.15.123:0/1217943504 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37a4014d90 con 0x7f37bc0f5410 2024-01-13T11:51:38.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.199+0000 7f37c117b700 1 -- 172.21.15.123:0/1217943504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37bc109b00 con 0x7f37bc0f5410 2024-01-13T11:51:38.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.200+0000 7f37b37fe700 1 -- 172.21.15.123:0/1217943504 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37a40095f0 con 0x7f37bc0f5410 2024-01-13T11:51:38.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.200+0000 7f37c117b700 1 -- 172.21.15.123:0/1217943504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37bc059100 con 0x7f37bc0f5410 2024-01-13T11:51:38.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.200+0000 7f37b37fe700 1 --2- 172.21.15.123:0/1217943504 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f37a802ff60 0x7f37a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:38.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.200+0000 7f37b37fe700 1 -- 172.21.15.123:0/1217943504 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f37a4040fe0 con 0x7f37bc0f5410 2024-01-13T11:51:38.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.200+0000 7f37ba59c700 1 --2- 172.21.15.123:0/1217943504 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f37a802ff60 0x7f37a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:38.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.200+0000 7f37ba59c700 1 --2- 172.21.15.123:0/1217943504 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f37a802ff60 0x7f37a8032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f37ac0096a0 tx=0x7f37ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:38.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.204+0000 7f37b37fe700 1 -- 172.21.15.123:0/1217943504 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37a401d7f0 con 0x7f37bc0f5410 2024-01-13T11:51:38.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.411+0000 7f37c117b700 1 -- 172.21.15.123:0/1217943504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37bc045ad0 con 0x7f37bc0f5410 2024-01-13T11:51:38.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.412+0000 7f37b37fe700 1 -- 172.21.15.123:0/1217943504 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f37a4010230 con 0x7f37bc0f5410 2024-01-13T11:51:38.401 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:38.401 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:38.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.413+0000 7f37c117b700 1 -- 172.21.15.123:0/1217943504 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f37a802ff60 msgr2=0x7f37a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:38.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.413+0000 7f37c117b700 1 --2- 172.21.15.123:0/1217943504 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f37a802ff60 0x7f37a8032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f37ac0096a0 tx=0x7f37ac006b40).stop 2024-01-13T11:51:38.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.413+0000 7f37c117b700 1 -- 172.21.15.123:0/1217943504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37bc0f5410 msgr2=0x7f37bc109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:38.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.413+0000 7f37c117b700 1 --2- 172.21.15.123:0/1217943504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37bc0f5410 0x7f37bc109450 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f37a4013510 tx=0x7f37a400b040).stop 2024-01-13T11:51:38.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.413+0000 7f37c117b700 1 -- 172.21.15.123:0/1217943504 shutdown_connections 2024-01-13T11:51:38.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.413+0000 7f37c117b700 1 --2- 172.21.15.123:0/1217943504 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f37a802ff60 0x7f37a8032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:38.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.413+0000 7f37c117b700 1 --2- 172.21.15.123:0/1217943504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37bc0f5410 0x7f37bc109450 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:38.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.413+0000 7f37c117b700 1 -- 172.21.15.123:0/1217943504 >> 172.21.15.123:0/1217943504 conn(0x7f37bc0f0b70 msgr2=0x7f37bc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:38.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.413+0000 7f37c117b700 1 -- 172.21.15.123:0/1217943504 shutdown_connections 2024-01-13T11:51:38.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:38.413+0000 7f37c117b700 1 -- 172.21.15.123:0/1217943504 wait complete. 2024-01-13T11:51:38.404 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:39.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:39 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:37.517836+0000 mgr.smithi059.hmyldj (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:39.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:39 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:38.413036+0000 mon.smithi059 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.123:0/1217943504' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:39.795 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:39.795 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:40.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:39.638243+0000 mon.smithi059 (mon.0) 149 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/mirror_snapshot_schedule"}]: dispatch 2024-01-13T11:51:40.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:39.640481+0000 mon.smithi059 (mon.0) 150 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/trash_purge_schedule"}]: dispatch 2024-01-13T11:51:40.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.979+0000 7f9feaa86700 1 -- 172.21.15.123:0/149138380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fe40f5410 msgr2=0x7f9fe40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:40.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.979+0000 7f9feaa86700 1 --2- 172.21.15.123:0/149138380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fe40f5410 0x7f9fe40f5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f9fd8009fb0 tx=0x7f9fd8009b30).stop 2024-01-13T11:51:40.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.980+0000 7f9feaa86700 1 -- 172.21.15.123:0/149138380 shutdown_connections 2024-01-13T11:51:40.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.980+0000 7f9feaa86700 1 --2- 172.21.15.123:0/149138380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fe40f5410 0x7f9fe40f5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:40.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.980+0000 7f9feaa86700 1 -- 172.21.15.123:0/149138380 >> 172.21.15.123:0/149138380 conn(0x7f9fe40f0b70 msgr2=0x7f9fe40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:40.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.980+0000 7f9feaa86700 1 -- 172.21.15.123:0/149138380 shutdown_connections 2024-01-13T11:51:40.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.980+0000 7f9feaa86700 1 -- 172.21.15.123:0/149138380 wait complete. 2024-01-13T11:51:40.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.980+0000 7f9feaa86700 1 Processor -- start 2024-01-13T11:51:40.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.980+0000 7f9feaa86700 1 -- start start 2024-01-13T11:51:40.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.981+0000 7f9feaa86700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fe40f5410 0x7f9fe40fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:40.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.981+0000 7f9feaa86700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe40fb820 con 0x7f9fe40f5410 2024-01-13T11:51:40.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.981+0000 7f9fe8822700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fe40f5410 0x7f9fe40fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:40.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.981+0000 7f9fe8822700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fe40f5410 0x7f9fe40fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:54334/0 (socket says 172.21.15.123:54334) 2024-01-13T11:51:40.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.981+0000 7f9fe8822700 1 -- 172.21.15.123:0/1091185601 learned_addr learned my addr 172.21.15.123:0/1091185601 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:40.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.981+0000 7f9fe8822700 1 -- 172.21.15.123:0/1091185601 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd8005040 con 0x7f9fe40f5410 2024-01-13T11:51:40.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.981+0000 7f9fe8822700 1 --2- 172.21.15.123:0/1091185601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fe40f5410 0x7f9fe40fd3a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f9fd8013510 tx=0x7f9fd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:40.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.981+0000 7f9fe17fa700 1 -- 172.21.15.123:0/1091185601 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fd8009320 con 0x7f9fe40f5410 2024-01-13T11:51:40.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.981+0000 7f9feaa86700 1 -- 172.21.15.123:0/1091185601 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe40fb9c0 con 0x7f9fe40f5410 2024-01-13T11:51:40.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.981+0000 7f9fe17fa700 1 -- 172.21.15.123:0/1091185601 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9fd8013f00 con 0x7f9fe40f5410 2024-01-13T11:51:40.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.981+0000 7f9fe17fa700 1 -- 172.21.15.123:0/1091185601 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fd8014d90 con 0x7f9fe40f5410 2024-01-13T11:51:40.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.981+0000 7f9feaa86700 1 -- 172.21.15.123:0/1091185601 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe40fbe50 con 0x7f9fe40f5410 2024-01-13T11:51:40.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.982+0000 7f9feaa86700 1 -- 172.21.15.123:0/1091185601 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fe4059100 con 0x7f9fe40f5410 2024-01-13T11:51:40.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.982+0000 7f9fe17fa700 1 -- 172.21.15.123:0/1091185601 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9fd8014420 con 0x7f9fe40f5410 2024-01-13T11:51:40.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.982+0000 7f9fe17fa700 1 --2- 172.21.15.123:0/1091185601 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9fd402ff60 0x7f9fd4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:40.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.982+0000 7f9fe17fa700 1 -- 172.21.15.123:0/1091185601 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9fd8041000 con 0x7f9fe40f5410 2024-01-13T11:51:40.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.983+0000 7f9fe3fff700 1 --2- 172.21.15.123:0/1091185601 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9fd402ff60 0x7f9fd4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:40.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.983+0000 7f9fe3fff700 1 --2- 172.21.15.123:0/1091185601 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9fd402ff60 0x7f9fd4032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9fd00096a0 tx=0x7f9fd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:40.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:40.987+0000 7f9fe17fa700 1 -- 172.21.15.123:0/1091185601 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9fd80095f0 con 0x7f9fe40f5410 2024-01-13T11:51:41.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:41.192+0000 7f9feaa86700 1 -- 172.21.15.123:0/1091185601 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9fe40fae60 con 0x7f9fe40f5410 2024-01-13T11:51:41.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:41.193+0000 7f9fe17fa700 1 -- 172.21.15.123:0/1091185601 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9fd801db70 con 0x7f9fe40f5410 2024-01-13T11:51:41.182 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:41.182 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:41.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:41.194+0000 7f9feaa86700 1 -- 172.21.15.123:0/1091185601 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9fd402ff60 msgr2=0x7f9fd4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:41.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:41.194+0000 7f9feaa86700 1 --2- 172.21.15.123:0/1091185601 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9fd402ff60 0x7f9fd4032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9fd00096a0 tx=0x7f9fd0006b40).stop 2024-01-13T11:51:41.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:41.194+0000 7f9feaa86700 1 -- 172.21.15.123:0/1091185601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fe40f5410 msgr2=0x7f9fe40fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:41.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:41.194+0000 7f9feaa86700 1 --2- 172.21.15.123:0/1091185601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fe40f5410 0x7f9fe40fd3a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f9fd8013510 tx=0x7f9fd800b040).stop 2024-01-13T11:51:41.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:41.194+0000 7f9feaa86700 1 -- 172.21.15.123:0/1091185601 shutdown_connections 2024-01-13T11:51:41.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:41.194+0000 7f9feaa86700 1 --2- 172.21.15.123:0/1091185601 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9fd402ff60 0x7f9fd4032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:41.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:41.194+0000 7f9feaa86700 1 --2- 172.21.15.123:0/1091185601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fe40f5410 0x7f9fe40fd3a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:41.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:41.194+0000 7f9feaa86700 1 -- 172.21.15.123:0/1091185601 >> 172.21.15.123:0/1091185601 conn(0x7f9fe40f0b70 msgr2=0x7f9fe40f1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:41.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:41.194+0000 7f9feaa86700 1 -- 172.21.15.123:0/1091185601 shutdown_connections 2024-01-13T11:51:41.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:41.194+0000 7f9feaa86700 1 -- 172.21.15.123:0/1091185601 wait complete. 2024-01-13T11:51:41.185 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:41.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:41 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:39.518133+0000 mgr.smithi059.hmyldj (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:41.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:41 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:41.193995+0000 mon.smithi059 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.123:0/1091185601' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:42.597 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:42.598 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:43.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:43 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:41.518476+0000 mgr.smithi059.hmyldj (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:43.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.729+0000 7fbc9336b700 1 -- 172.21.15.123:0/4126379769 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc8c0f5410 msgr2=0x7fbc8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:43.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.729+0000 7fbc9336b700 1 --2- 172.21.15.123:0/4126379769 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc8c0f5410 0x7fbc8c0f5830 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fbc7c009fb0 tx=0x7fbc7c009b30).stop 2024-01-13T11:51:43.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.729+0000 7fbc9336b700 1 -- 172.21.15.123:0/4126379769 shutdown_connections 2024-01-13T11:51:43.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.729+0000 7fbc9336b700 1 --2- 172.21.15.123:0/4126379769 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc8c0f5410 0x7fbc8c0f5830 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:43.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.729+0000 7fbc9336b700 1 -- 172.21.15.123:0/4126379769 >> 172.21.15.123:0/4126379769 conn(0x7fbc8c0f0b70 msgr2=0x7fbc8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:43.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.729+0000 7fbc9336b700 1 -- 172.21.15.123:0/4126379769 shutdown_connections 2024-01-13T11:51:43.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.729+0000 7fbc9336b700 1 -- 172.21.15.123:0/4126379769 wait complete. 2024-01-13T11:51:43.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.730+0000 7fbc9336b700 1 Processor -- start 2024-01-13T11:51:43.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.730+0000 7fbc9336b700 1 -- start start 2024-01-13T11:51:43.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.730+0000 7fbc9336b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc8c0f5410 0x7fbc8c10b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:43.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.730+0000 7fbc9336b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c107240 con 0x7fbc8c0f5410 2024-01-13T11:51:43.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.730+0000 7fbc91107700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc8c0f5410 0x7fbc8c10b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:43.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.730+0000 7fbc91107700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc8c0f5410 0x7fbc8c10b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:54360/0 (socket says 172.21.15.123:54360) 2024-01-13T11:51:43.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.730+0000 7fbc91107700 1 -- 172.21.15.123:0/3264673250 learned_addr learned my addr 172.21.15.123:0/3264673250 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:43.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.731+0000 7fbc91107700 1 -- 172.21.15.123:0/3264673250 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7c005040 con 0x7fbc8c0f5410 2024-01-13T11:51:43.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.731+0000 7fbc91107700 1 --2- 172.21.15.123:0/3264673250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc8c0f5410 0x7fbc8c10b850 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fbc7c013510 tx=0x7fbc7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:43.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.731+0000 7fbc81ffb700 1 -- 172.21.15.123:0/3264673250 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc7c009320 con 0x7fbc8c0f5410 2024-01-13T11:51:43.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.731+0000 7fbc9336b700 1 -- 172.21.15.123:0/3264673250 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8c060220 con 0x7fbc8c0f5410 2024-01-13T11:51:43.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.731+0000 7fbc81ffb700 1 -- 172.21.15.123:0/3264673250 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc7c013f00 con 0x7fbc8c0f5410 2024-01-13T11:51:43.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.731+0000 7fbc81ffb700 1 -- 172.21.15.123:0/3264673250 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc7c014d90 con 0x7fbc8c0f5410 2024-01-13T11:51:43.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.731+0000 7fbc9336b700 1 -- 172.21.15.123:0/3264673250 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8c10bd40 con 0x7fbc8c0f5410 2024-01-13T11:51:43.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.732+0000 7fbc81ffb700 1 -- 172.21.15.123:0/3264673250 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc7c0095f0 con 0x7fbc8c0f5410 2024-01-13T11:51:43.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.732+0000 7fbc9336b700 1 -- 172.21.15.123:0/3264673250 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc8c045ad0 con 0x7fbc8c0f5410 2024-01-13T11:51:43.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.732+0000 7fbc81ffb700 1 --2- 172.21.15.123:0/3264673250 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbc7802ffb0 0x7fbc78032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:43.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.732+0000 7fbc81ffb700 1 -- 172.21.15.123:0/3264673250 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc7c041080 con 0x7fbc8c0f5410 2024-01-13T11:51:43.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.733+0000 7fbc90906700 1 --2- 172.21.15.123:0/3264673250 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbc7802ffb0 0x7fbc78032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:43.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.733+0000 7fbc90906700 1 --2- 172.21.15.123:0/3264673250 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbc7802ffb0 0x7fbc78032470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fbc880096a0 tx=0x7fbc88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:43.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.737+0000 7fbc81ffb700 1 -- 172.21.15.123:0/3264673250 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc7c06d2b0 con 0x7fbc8c0f5410 2024-01-13T11:51:43.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.945+0000 7fbc9336b700 1 -- 172.21.15.123:0/3264673250 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc8c0fae60 con 0x7fbc8c0f5410 2024-01-13T11:51:43.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.945+0000 7fbc81ffb700 1 -- 172.21.15.123:0/3264673250 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbc7c014420 con 0x7fbc8c0f5410 2024-01-13T11:51:43.935 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:43.935 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:43.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.947+0000 7fbc9336b700 1 -- 172.21.15.123:0/3264673250 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbc7802ffb0 msgr2=0x7fbc78032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:43.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.947+0000 7fbc9336b700 1 --2- 172.21.15.123:0/3264673250 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbc7802ffb0 0x7fbc78032470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fbc880096a0 tx=0x7fbc88006b40).stop 2024-01-13T11:51:43.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.947+0000 7fbc9336b700 1 -- 172.21.15.123:0/3264673250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc8c0f5410 msgr2=0x7fbc8c10b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:43.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.947+0000 7fbc9336b700 1 --2- 172.21.15.123:0/3264673250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc8c0f5410 0x7fbc8c10b850 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fbc7c013510 tx=0x7fbc7c00b040).stop 2024-01-13T11:51:43.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.947+0000 7fbc9336b700 1 -- 172.21.15.123:0/3264673250 shutdown_connections 2024-01-13T11:51:43.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.947+0000 7fbc9336b700 1 --2- 172.21.15.123:0/3264673250 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbc7802ffb0 0x7fbc78032470 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:43.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.947+0000 7fbc9336b700 1 --2- 172.21.15.123:0/3264673250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc8c0f5410 0x7fbc8c10b850 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:43.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.947+0000 7fbc9336b700 1 -- 172.21.15.123:0/3264673250 >> 172.21.15.123:0/3264673250 conn(0x7fbc8c0f0b70 msgr2=0x7fbc8c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:43.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.947+0000 7fbc9336b700 1 -- 172.21.15.123:0/3264673250 shutdown_connections 2024-01-13T11:51:43.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:43.947+0000 7fbc9336b700 1 -- 172.21.15.123:0/3264673250 wait complete. 2024-01-13T11:51:43.937 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:44.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:44 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:43.946539+0000 mon.smithi059 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.123:0/3264673250' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:45.331 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:45.332 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:45.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:45 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:43.518723+0000 mgr.smithi059.hmyldj (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:46.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.517+0000 7f270a06a700 1 -- 172.21.15.123:0/1854134163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27040f5410 msgr2=0x7f27040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:46.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.517+0000 7f270a06a700 1 --2- 172.21.15.123:0/1854134163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27040f5410 0x7f27040f5830 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f26ec009fb0 tx=0x7f26ec009b30).stop 2024-01-13T11:51:46.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.517+0000 7f270a06a700 1 -- 172.21.15.123:0/1854134163 shutdown_connections 2024-01-13T11:51:46.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.517+0000 7f270a06a700 1 --2- 172.21.15.123:0/1854134163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27040f5410 0x7f27040f5830 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:46.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.517+0000 7f270a06a700 1 -- 172.21.15.123:0/1854134163 >> 172.21.15.123:0/1854134163 conn(0x7f27040f0b70 msgr2=0x7f27040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:46.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.517+0000 7f270a06a700 1 -- 172.21.15.123:0/1854134163 shutdown_connections 2024-01-13T11:51:46.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.517+0000 7f270a06a700 1 -- 172.21.15.123:0/1854134163 wait complete. 2024-01-13T11:51:46.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.517+0000 7f270a06a700 1 Processor -- start 2024-01-13T11:51:46.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.517+0000 7f270a06a700 1 -- start start 2024-01-13T11:51:46.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.518+0000 7f270a06a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27040f5410 0x7f2704106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:46.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.518+0000 7f270a06a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2704106c20 con 0x7f27040f5410 2024-01-13T11:51:46.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.518+0000 7f27037fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27040f5410 0x7f2704106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:46.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.518+0000 7f27037fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27040f5410 0x7f2704106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:47258/0 (socket says 172.21.15.123:47258) 2024-01-13T11:51:46.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.518+0000 7f27037fe700 1 -- 172.21.15.123:0/1764213874 learned_addr learned my addr 172.21.15.123:0/1764213874 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:46.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.518+0000 7f27037fe700 1 -- 172.21.15.123:0/1764213874 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec005040 con 0x7f27040f5410 2024-01-13T11:51:46.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.518+0000 7f27037fe700 1 --2- 172.21.15.123:0/1764213874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27040f5410 0x7f2704106730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f26ec013510 tx=0x7f26ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:46.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.518+0000 7f26fbfff700 1 -- 172.21.15.123:0/1764213874 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26ec009320 con 0x7f27040f5410 2024-01-13T11:51:46.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.518+0000 7f270a06a700 1 -- 172.21.15.123:0/1764213874 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2704106e20 con 0x7f27040f5410 2024-01-13T11:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.518+0000 7f26fbfff700 1 -- 172.21.15.123:0/1764213874 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f26ec013f00 con 0x7f27040f5410 2024-01-13T11:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.518+0000 7f26fbfff700 1 -- 172.21.15.123:0/1764213874 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26ec014d90 con 0x7f27040f5410 2024-01-13T11:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.518+0000 7f270a06a700 1 -- 172.21.15.123:0/1764213874 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27041072b0 con 0x7f27040f5410 2024-01-13T11:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.519+0000 7f26fbfff700 1 -- 172.21.15.123:0/1764213874 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26ec014420 con 0x7f27040f5410 2024-01-13T11:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.519+0000 7f270a06a700 1 -- 172.21.15.123:0/1764213874 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2704045ad0 con 0x7f27040f5410 2024-01-13T11:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.519+0000 7f26fbfff700 1 --2- 172.21.15.123:0/1764213874 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f26f002ff60 0x7f26f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.519+0000 7f26fbfff700 1 -- 172.21.15.123:0/1764213874 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f26ec040fb0 con 0x7f27040f5410 2024-01-13T11:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.520+0000 7f2702ffd700 1 --2- 172.21.15.123:0/1764213874 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f26f002ff60 0x7f26f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.520+0000 7f2702ffd700 1 --2- 172.21.15.123:0/1764213874 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f26f002ff60 0x7f26f0032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f26f40096a0 tx=0x7f26f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:46.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.524+0000 7f26fbfff700 1 -- 172.21.15.123:0/1764213874 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26ec0095f0 con 0x7f27040f5410 2024-01-13T11:51:46.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.732+0000 7f270a06a700 1 -- 172.21.15.123:0/1764213874 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f27040fbf90 con 0x7f27040f5410 2024-01-13T11:51:46.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.733+0000 7f26fbfff700 1 -- 172.21.15.123:0/1764213874 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f26ec01db70 con 0x7f27040f5410 2024-01-13T11:51:46.722 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:46.723 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:46.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.735+0000 7f270a06a700 1 -- 172.21.15.123:0/1764213874 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f26f002ff60 msgr2=0x7f26f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:46.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.735+0000 7f270a06a700 1 --2- 172.21.15.123:0/1764213874 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f26f002ff60 0x7f26f0032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f26f40096a0 tx=0x7f26f4006b40).stop 2024-01-13T11:51:46.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.735+0000 7f270a06a700 1 -- 172.21.15.123:0/1764213874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27040f5410 msgr2=0x7f2704106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:46.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.735+0000 7f270a06a700 1 --2- 172.21.15.123:0/1764213874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27040f5410 0x7f2704106730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f26ec013510 tx=0x7f26ec00b040).stop 2024-01-13T11:51:46.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.735+0000 7f270a06a700 1 -- 172.21.15.123:0/1764213874 shutdown_connections 2024-01-13T11:51:46.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.735+0000 7f270a06a700 1 --2- 172.21.15.123:0/1764213874 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f26f002ff60 0x7f26f0032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:46.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.735+0000 7f270a06a700 1 --2- 172.21.15.123:0/1764213874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27040f5410 0x7f2704106730 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:46.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.735+0000 7f270a06a700 1 -- 172.21.15.123:0/1764213874 >> 172.21.15.123:0/1764213874 conn(0x7f27040f0b70 msgr2=0x7f27040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:46.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.735+0000 7f270a06a700 1 -- 172.21.15.123:0/1764213874 shutdown_connections 2024-01-13T11:51:46.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:46.735+0000 7f270a06a700 1 -- 172.21.15.123:0/1764213874 wait complete. 2024-01-13T11:51:46.725 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:47.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:47 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:45.518998+0000 mgr.smithi059.hmyldj (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:47.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:47 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:46.734397+0000 mon.smithi059 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.123:0/1764213874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:48.123 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:48.123 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:49.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.283+0000 7f9e3ea20700 1 -- 172.21.15.123:0/181323448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f51a0 msgr2=0x7f9e380f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:49.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.283+0000 7f9e3ea20700 1 --2- 172.21.15.123:0/181323448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f51a0 0x7f9e380f55c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f9e28009fb0 tx=0x7f9e28009b30).stop 2024-01-13T11:51:49.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.283+0000 7f9e3ea20700 1 -- 172.21.15.123:0/181323448 shutdown_connections 2024-01-13T11:51:49.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.283+0000 7f9e3ea20700 1 --2- 172.21.15.123:0/181323448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f51a0 0x7f9e380f55c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:49.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.283+0000 7f9e3ea20700 1 -- 172.21.15.123:0/181323448 >> 172.21.15.123:0/181323448 conn(0x7f9e380f09a0 msgr2=0x7f9e380f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:49.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.284+0000 7f9e3ea20700 1 -- 172.21.15.123:0/181323448 shutdown_connections 2024-01-13T11:51:49.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.284+0000 7f9e3ea20700 1 -- 172.21.15.123:0/181323448 wait complete. 2024-01-13T11:51:49.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.284+0000 7f9e3ea20700 1 Processor -- start 2024-01-13T11:51:49.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.284+0000 7f9e3ea20700 1 -- start start 2024-01-13T11:51:49.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.284+0000 7f9e3ea20700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f51a0 0x7f9e381091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:49.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.284+0000 7f9e3ea20700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e38060240 con 0x7f9e380f51a0 2024-01-13T11:51:49.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.284+0000 7f9e3da1e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f51a0 0x7f9e381091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:49.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.284+0000 7f9e3da1e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f51a0 0x7f9e381091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:47276/0 (socket says 172.21.15.123:47276) 2024-01-13T11:51:49.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.284+0000 7f9e3da1e700 1 -- 172.21.15.123:0/1447627291 learned_addr learned my addr 172.21.15.123:0/1447627291 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:49.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.285+0000 7f9e3da1e700 1 -- 172.21.15.123:0/1447627291 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e28005040 con 0x7f9e380f51a0 2024-01-13T11:51:49.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.285+0000 7f9e3da1e700 1 --2- 172.21.15.123:0/1447627291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f51a0 0x7f9e381091b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f9e28013510 tx=0x7f9e2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:49.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.285+0000 7f9e2e7fc700 1 -- 172.21.15.123:0/1447627291 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e28009320 con 0x7f9e380f51a0 2024-01-13T11:51:49.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.285+0000 7f9e3ea20700 1 -- 172.21.15.123:0/1447627291 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e38060440 con 0x7f9e380f51a0 2024-01-13T11:51:49.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.285+0000 7f9e2e7fc700 1 -- 172.21.15.123:0/1447627291 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e28013f00 con 0x7f9e380f51a0 2024-01-13T11:51:49.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.285+0000 7f9e2e7fc700 1 -- 172.21.15.123:0/1447627291 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e28014dc0 con 0x7f9e380f51a0 2024-01-13T11:51:49.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.285+0000 7f9e3ea20700 1 -- 172.21.15.123:0/1447627291 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e38109860 con 0x7f9e380f51a0 2024-01-13T11:51:49.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.286+0000 7f9e2e7fc700 1 -- 172.21.15.123:0/1447627291 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e28014420 con 0x7f9e380f51a0 2024-01-13T11:51:49.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.286+0000 7f9e3ea20700 1 -- 172.21.15.123:0/1447627291 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e38045ad0 con 0x7f9e380f51a0 2024-01-13T11:51:49.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.286+0000 7f9e2e7fc700 1 --2- 172.21.15.123:0/1447627291 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9e2402ff60 0x7f9e24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:49.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.286+0000 7f9e2e7fc700 1 -- 172.21.15.123:0/1447627291 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e28041110 con 0x7f9e380f51a0 2024-01-13T11:51:49.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.286+0000 7f9e3d21d700 1 --2- 172.21.15.123:0/1447627291 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9e2402ff60 0x7f9e24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:49.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.287+0000 7f9e3d21d700 1 --2- 172.21.15.123:0/1447627291 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9e2402ff60 0x7f9e24032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f9e340096a0 tx=0x7f9e34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:49.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.291+0000 7f9e2e7fc700 1 -- 172.21.15.123:0/1447627291 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e280095f0 con 0x7f9e380f51a0 2024-01-13T11:51:49.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.499+0000 7f9e3ea20700 1 -- 172.21.15.123:0/1447627291 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e3810a080 con 0x7f9e380f51a0 2024-01-13T11:51:49.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.500+0000 7f9e2e7fc700 1 -- 172.21.15.123:0/1447627291 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9e2801cb10 con 0x7f9e380f51a0 2024-01-13T11:51:49.488 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:49.489 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:49.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.502+0000 7f9e3ea20700 1 -- 172.21.15.123:0/1447627291 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9e2402ff60 msgr2=0x7f9e24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:49.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.502+0000 7f9e3ea20700 1 --2- 172.21.15.123:0/1447627291 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9e2402ff60 0x7f9e24032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f9e340096a0 tx=0x7f9e34006b40).stop 2024-01-13T11:51:49.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.502+0000 7f9e3ea20700 1 -- 172.21.15.123:0/1447627291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f51a0 msgr2=0x7f9e381091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:49.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.502+0000 7f9e3ea20700 1 --2- 172.21.15.123:0/1447627291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f51a0 0x7f9e381091b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f9e28013510 tx=0x7f9e2800b040).stop 2024-01-13T11:51:49.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.502+0000 7f9e3ea20700 1 -- 172.21.15.123:0/1447627291 shutdown_connections 2024-01-13T11:51:49.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.502+0000 7f9e3ea20700 1 --2- 172.21.15.123:0/1447627291 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9e2402ff60 0x7f9e24032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:49.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.502+0000 7f9e3ea20700 1 --2- 172.21.15.123:0/1447627291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f51a0 0x7f9e381091b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:49.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.502+0000 7f9e3ea20700 1 -- 172.21.15.123:0/1447627291 >> 172.21.15.123:0/1447627291 conn(0x7f9e380f09a0 msgr2=0x7f9e380f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:49.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.502+0000 7f9e3ea20700 1 -- 172.21.15.123:0/1447627291 shutdown_connections 2024-01-13T11:51:49.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:49.502+0000 7f9e3ea20700 1 -- 172.21.15.123:0/1447627291 wait complete. 2024-01-13T11:51:49.491 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:49.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:49 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:47.519326+0000 mgr.smithi059.hmyldj (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:50.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:50 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:49.500213+0000 mon.smithi059 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.123:0/1447627291' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:50.874 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:50.875 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:51.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:49.519621+0000 mgr.smithi059.hmyldj (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:52.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.036+0000 7f47c2558700 1 -- 172.21.15.123:0/2667149066 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47bc0f6510 msgr2=0x7f47bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:52.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.036+0000 7f47c2558700 1 --2- 172.21.15.123:0/2667149066 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47bc0f6510 0x7f47bc0f6930 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f47a4009fb0 tx=0x7f47a4009b30).stop 2024-01-13T11:51:52.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.036+0000 7f47c2558700 1 -- 172.21.15.123:0/2667149066 shutdown_connections 2024-01-13T11:51:52.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.036+0000 7f47c2558700 1 --2- 172.21.15.123:0/2667149066 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47bc0f6510 0x7f47bc0f6930 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:52.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.036+0000 7f47c2558700 1 -- 172.21.15.123:0/2667149066 >> 172.21.15.123:0/2667149066 conn(0x7f47bc0f1c70 msgr2=0x7f47bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:52.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.036+0000 7f47c2558700 1 -- 172.21.15.123:0/2667149066 shutdown_connections 2024-01-13T11:51:52.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.036+0000 7f47c2558700 1 -- 172.21.15.123:0/2667149066 wait complete. 2024-01-13T11:51:52.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.037+0000 7f47c2558700 1 Processor -- start 2024-01-13T11:51:52.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.037+0000 7f47c2558700 1 -- start start 2024-01-13T11:51:52.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.037+0000 7f47c2558700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47bc0f6510 0x7f47bc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:52.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.037+0000 7f47c2558700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc106c20 con 0x7f47bc0f6510 2024-01-13T11:51:52.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.037+0000 7f47bbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47bc0f6510 0x7f47bc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:52.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.037+0000 7f47bbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47bc0f6510 0x7f47bc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:47296/0 (socket says 172.21.15.123:47296) 2024-01-13T11:51:52.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.037+0000 7f47bbfff700 1 -- 172.21.15.123:0/1996938028 learned_addr learned my addr 172.21.15.123:0/1996938028 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:52.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.038+0000 7f47bbfff700 1 -- 172.21.15.123:0/1996938028 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a4005040 con 0x7f47bc0f6510 2024-01-13T11:51:52.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.038+0000 7f47bbfff700 1 --2- 172.21.15.123:0/1996938028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47bc0f6510 0x7f47bc106730 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f47a4013510 tx=0x7f47a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:52.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.038+0000 7f47b8ff9700 1 -- 172.21.15.123:0/1996938028 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47a4009320 con 0x7f47bc0f6510 2024-01-13T11:51:52.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.038+0000 7f47c2558700 1 -- 172.21.15.123:0/1996938028 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47bc106e20 con 0x7f47bc0f6510 2024-01-13T11:51:52.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.038+0000 7f47b8ff9700 1 -- 172.21.15.123:0/1996938028 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47a4013f00 con 0x7f47bc0f6510 2024-01-13T11:51:52.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.038+0000 7f47b8ff9700 1 -- 172.21.15.123:0/1996938028 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47a4014d90 con 0x7f47bc0f6510 2024-01-13T11:51:52.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.038+0000 7f47c2558700 1 -- 172.21.15.123:0/1996938028 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47bc1072b0 con 0x7f47bc0f6510 2024-01-13T11:51:52.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.039+0000 7f47b8ff9700 1 -- 172.21.15.123:0/1996938028 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f47a40263a0 con 0x7f47bc0f6510 2024-01-13T11:51:52.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.039+0000 7f47b8ff9700 1 --2- 172.21.15.123:0/1996938028 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f47a802ff60 0x7f47a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:52.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.039+0000 7f47b8ff9700 1 -- 172.21.15.123:0/1996938028 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f47a40410d0 con 0x7f47bc0f6510 2024-01-13T11:51:52.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.039+0000 7f47c2558700 1 -- 172.21.15.123:0/1996938028 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47bc045ad0 con 0x7f47bc0f6510 2024-01-13T11:51:52.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.039+0000 7f47bb7fe700 1 --2- 172.21.15.123:0/1996938028 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f47a802ff60 0x7f47a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:52.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.040+0000 7f47bb7fe700 1 --2- 172.21.15.123:0/1996938028 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f47a802ff60 0x7f47a8032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f47ac0096a0 tx=0x7f47ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:52.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.044+0000 7f47b8ff9700 1 -- 172.21.15.123:0/1996938028 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47a4012920 con 0x7f47bc0f6510 2024-01-13T11:51:52.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.252+0000 7f47c2558700 1 -- 172.21.15.123:0/1996938028 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47bc0fc050 con 0x7f47bc0f6510 2024-01-13T11:51:52.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.253+0000 7f47b8ff9700 1 -- 172.21.15.123:0/1996938028 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f47a402b020 con 0x7f47bc0f6510 2024-01-13T11:51:52.241 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:52.241 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:52.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.254+0000 7f47c2558700 1 -- 172.21.15.123:0/1996938028 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f47a802ff60 msgr2=0x7f47a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:52.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.254+0000 7f47c2558700 1 --2- 172.21.15.123:0/1996938028 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f47a802ff60 0x7f47a8032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f47ac0096a0 tx=0x7f47ac006b40).stop 2024-01-13T11:51:52.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.254+0000 7f47c2558700 1 -- 172.21.15.123:0/1996938028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47bc0f6510 msgr2=0x7f47bc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:52.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.254+0000 7f47c2558700 1 --2- 172.21.15.123:0/1996938028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47bc0f6510 0x7f47bc106730 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f47a4013510 tx=0x7f47a400b040).stop 2024-01-13T11:51:52.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.254+0000 7f47c2558700 1 -- 172.21.15.123:0/1996938028 shutdown_connections 2024-01-13T11:51:52.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.254+0000 7f47c2558700 1 --2- 172.21.15.123:0/1996938028 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f47a802ff60 0x7f47a8032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:52.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.254+0000 7f47c2558700 1 --2- 172.21.15.123:0/1996938028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47bc0f6510 0x7f47bc106730 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:52.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.254+0000 7f47c2558700 1 -- 172.21.15.123:0/1996938028 >> 172.21.15.123:0/1996938028 conn(0x7f47bc0f1c70 msgr2=0x7f47bc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:52.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.254+0000 7f47c2558700 1 -- 172.21.15.123:0/1996938028 shutdown_connections 2024-01-13T11:51:52.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:52.254+0000 7f47c2558700 1 -- 172.21.15.123:0/1996938028 wait complete. 2024-01-13T11:51:52.243 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:52.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:51.519961+0000 mgr.smithi059.hmyldj (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:53.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:53 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:52.252910+0000 mon.smithi059 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.123:0/1996938028' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:53.668 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:53.668 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:54.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:54 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:53.520289+0000 mgr.smithi059.hmyldj (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:54.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.860+0000 7f6ed99df700 1 -- 172.21.15.123:0/2054166393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ed40f5410 msgr2=0x7f6ed40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:54.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.860+0000 7f6ed99df700 1 --2- 172.21.15.123:0/2054166393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ed40f5410 0x7f6ed40f5830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f6ec8009fb0 tx=0x7f6ec8009b30).stop 2024-01-13T11:51:54.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.860+0000 7f6ed99df700 1 -- 172.21.15.123:0/2054166393 shutdown_connections 2024-01-13T11:51:54.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.860+0000 7f6ed99df700 1 --2- 172.21.15.123:0/2054166393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ed40f5410 0x7f6ed40f5830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:54.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.860+0000 7f6ed99df700 1 -- 172.21.15.123:0/2054166393 >> 172.21.15.123:0/2054166393 conn(0x7f6ed40f0b70 msgr2=0x7f6ed40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:54.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.860+0000 7f6ed99df700 1 -- 172.21.15.123:0/2054166393 shutdown_connections 2024-01-13T11:51:54.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.860+0000 7f6ed99df700 1 -- 172.21.15.123:0/2054166393 wait complete. 2024-01-13T11:51:54.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.861+0000 7f6ed99df700 1 Processor -- start 2024-01-13T11:51:54.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.861+0000 7f6ed99df700 1 -- start start 2024-01-13T11:51:54.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.861+0000 7f6ed99df700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ed40f5410 0x7f6ed41093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:54.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.861+0000 7f6ed99df700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed4060280 con 0x7f6ed40f5410 2024-01-13T11:51:54.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.861+0000 7f6ed2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ed40f5410 0x7f6ed41093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:54.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.861+0000 7f6ed2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ed40f5410 0x7f6ed41093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:47316/0 (socket says 172.21.15.123:47316) 2024-01-13T11:51:54.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.861+0000 7f6ed2ffd700 1 -- 172.21.15.123:0/1961330538 learned_addr learned my addr 172.21.15.123:0/1961330538 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:54.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.862+0000 7f6ed2ffd700 1 -- 172.21.15.123:0/1961330538 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec8005040 con 0x7f6ed40f5410 2024-01-13T11:51:54.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.862+0000 7f6ed2ffd700 1 --2- 172.21.15.123:0/1961330538 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ed40f5410 0x7f6ed41093c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f6ec8013510 tx=0x7f6ec800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:54.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.862+0000 7f6ebb7fe700 1 -- 172.21.15.123:0/1961330538 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ec8009320 con 0x7f6ed40f5410 2024-01-13T11:51:54.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.862+0000 7f6ed99df700 1 -- 172.21.15.123:0/1961330538 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed4060480 con 0x7f6ed40f5410 2024-01-13T11:51:54.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.862+0000 7f6ebb7fe700 1 -- 172.21.15.123:0/1961330538 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ec8013f00 con 0x7f6ed40f5410 2024-01-13T11:51:54.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.862+0000 7f6ebb7fe700 1 -- 172.21.15.123:0/1961330538 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ec8014d90 con 0x7f6ed40f5410 2024-01-13T11:51:54.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.862+0000 7f6ed99df700 1 -- 172.21.15.123:0/1961330538 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed4109af0 con 0x7f6ed40f5410 2024-01-13T11:51:54.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.863+0000 7f6ebb7fe700 1 -- 172.21.15.123:0/1961330538 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6ec8014420 con 0x7f6ed40f5410 2024-01-13T11:51:54.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.863+0000 7f6ed99df700 1 -- 172.21.15.123:0/1961330538 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed4045ad0 con 0x7f6ed40f5410 2024-01-13T11:51:54.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.863+0000 7f6ebb7fe700 1 --2- 172.21.15.123:0/1961330538 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6ebc02ff60 0x7f6ebc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:54.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.863+0000 7f6ebb7fe700 1 -- 172.21.15.123:0/1961330538 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6ec8041070 con 0x7f6ed40f5410 2024-01-13T11:51:54.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.863+0000 7f6ed27fc700 1 --2- 172.21.15.123:0/1961330538 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6ebc02ff60 0x7f6ebc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:54.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.864+0000 7f6ed27fc700 1 --2- 172.21.15.123:0/1961330538 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6ebc02ff60 0x7f6ebc032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f6ec40096a0 tx=0x7f6ec4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:54.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:54.868+0000 7f6ebb7fe700 1 -- 172.21.15.123:0/1961330538 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ec80095f0 con 0x7f6ed40f5410 2024-01-13T11:51:55.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:55.076+0000 7f6ed99df700 1 -- 172.21.15.123:0/1961330538 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ed410a310 con 0x7f6ed40f5410 2024-01-13T11:51:55.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:55.076+0000 7f6ebb7fe700 1 -- 172.21.15.123:0/1961330538 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6ec801db70 con 0x7f6ed40f5410 2024-01-13T11:51:55.066 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:55.066 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:55.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:55.078+0000 7f6ed99df700 1 -- 172.21.15.123:0/1961330538 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6ebc02ff60 msgr2=0x7f6ebc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:55.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:55.078+0000 7f6ed99df700 1 --2- 172.21.15.123:0/1961330538 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6ebc02ff60 0x7f6ebc032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f6ec40096a0 tx=0x7f6ec4006b40).stop 2024-01-13T11:51:55.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:55.078+0000 7f6ed99df700 1 -- 172.21.15.123:0/1961330538 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ed40f5410 msgr2=0x7f6ed41093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:55.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:55.078+0000 7f6ed99df700 1 --2- 172.21.15.123:0/1961330538 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ed40f5410 0x7f6ed41093c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f6ec8013510 tx=0x7f6ec800b040).stop 2024-01-13T11:51:55.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:55.078+0000 7f6ed99df700 1 -- 172.21.15.123:0/1961330538 shutdown_connections 2024-01-13T11:51:55.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:55.078+0000 7f6ed99df700 1 --2- 172.21.15.123:0/1961330538 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6ebc02ff60 0x7f6ebc032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:55.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:55.078+0000 7f6ed99df700 1 --2- 172.21.15.123:0/1961330538 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ed40f5410 0x7f6ed41093c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:55.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:55.078+0000 7f6ed99df700 1 -- 172.21.15.123:0/1961330538 >> 172.21.15.123:0/1961330538 conn(0x7f6ed40f0b70 msgr2=0x7f6ed40f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:55.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:55.078+0000 7f6ed99df700 1 -- 172.21.15.123:0/1961330538 shutdown_connections 2024-01-13T11:51:55.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:55.078+0000 7f6ed99df700 1 -- 172.21.15.123:0/1961330538 wait complete. 2024-01-13T11:51:55.068 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:55.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:55 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:55.077675+0000 mon.smithi059 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.123:0/1961330538' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:56.479 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:56.479 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:57.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:55.520572+0000 mgr.smithi059.hmyldj (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:57.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.652+0000 7f165ab67700 1 -- 172.21.15.123:0/3710701475 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1654060220 msgr2=0x7f16540f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:57.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.652+0000 7f165ab67700 1 --2- 172.21.15.123:0/3710701475 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1654060220 0x7f16540f9150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f1648009fb0 tx=0x7f1648009b30).stop 2024-01-13T11:51:57.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.652+0000 7f165ab67700 1 -- 172.21.15.123:0/3710701475 shutdown_connections 2024-01-13T11:51:57.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.652+0000 7f165ab67700 1 --2- 172.21.15.123:0/3710701475 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1654060220 0x7f16540f9150 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:57.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.652+0000 7f165ab67700 1 -- 172.21.15.123:0/3710701475 >> 172.21.15.123:0/3710701475 conn(0x7f16540f0b70 msgr2=0x7f16540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:57.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.652+0000 7f165ab67700 1 -- 172.21.15.123:0/3710701475 shutdown_connections 2024-01-13T11:51:57.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.652+0000 7f165ab67700 1 -- 172.21.15.123:0/3710701475 wait complete. 2024-01-13T11:51:57.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.653+0000 7f165ab67700 1 Processor -- start 2024-01-13T11:51:57.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.653+0000 7f165ab67700 1 -- start start 2024-01-13T11:51:57.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.653+0000 7f165ab67700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1654060220 0x7f16541066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:57.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.653+0000 7f165ab67700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1654106b90 con 0x7f1654060220 2024-01-13T11:51:57.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.653+0000 7f1658903700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1654060220 0x7f16541066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:57.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.653+0000 7f1658903700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1654060220 0x7f16541066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:55320/0 (socket says 172.21.15.123:55320) 2024-01-13T11:51:57.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.653+0000 7f1658903700 1 -- 172.21.15.123:0/543056101 learned_addr learned my addr 172.21.15.123:0/543056101 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:51:57.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.654+0000 7f1658903700 1 -- 172.21.15.123:0/543056101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1648005040 con 0x7f1654060220 2024-01-13T11:51:57.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.654+0000 7f1658903700 1 --2- 172.21.15.123:0/543056101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1654060220 0x7f16541066a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f1648013510 tx=0x7f164800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:57.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.654+0000 7f16517fa700 1 -- 172.21.15.123:0/543056101 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1648009320 con 0x7f1654060220 2024-01-13T11:51:57.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.654+0000 7f165ab67700 1 -- 172.21.15.123:0/543056101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1654106d90 con 0x7f1654060220 2024-01-13T11:51:57.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.654+0000 7f16517fa700 1 -- 172.21.15.123:0/543056101 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1648013f00 con 0x7f1654060220 2024-01-13T11:51:57.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.654+0000 7f16517fa700 1 -- 172.21.15.123:0/543056101 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1648014dc0 con 0x7f1654060220 2024-01-13T11:51:57.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.654+0000 7f165ab67700 1 -- 172.21.15.123:0/543056101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1654107220 con 0x7f1654060220 2024-01-13T11:51:57.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.655+0000 7f16517fa700 1 -- 172.21.15.123:0/543056101 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1648014420 con 0x7f1654060220 2024-01-13T11:51:57.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.655+0000 7f165ab67700 1 -- 172.21.15.123:0/543056101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1654045ad0 con 0x7f1654060220 2024-01-13T11:51:57.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.655+0000 7f16517fa700 1 --2- 172.21.15.123:0/543056101 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f164402ffb0 0x7f1644032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:51:57.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.655+0000 7f16517fa700 1 -- 172.21.15.123:0/543056101 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1648040ff0 con 0x7f1654060220 2024-01-13T11:51:57.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.655+0000 7f1653fff700 1 --2- 172.21.15.123:0/543056101 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f164402ffb0 0x7f1644032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:51:57.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.655+0000 7f1653fff700 1 --2- 172.21.15.123:0/543056101 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f164402ffb0 0x7f1644032470 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f16400096a0 tx=0x7f1640006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:51:57.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.660+0000 7f16517fa700 1 -- 172.21.15.123:0/543056101 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f164806d2b0 con 0x7f1654060220 2024-01-13T11:51:57.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.866+0000 7f165ab67700 1 -- 172.21.15.123:0/543056101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16541080d0 con 0x7f1654060220 2024-01-13T11:51:57.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.867+0000 7f16517fa700 1 -- 172.21.15.123:0/543056101 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f16480095f0 con 0x7f1654060220 2024-01-13T11:51:57.857 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:51:57.857 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:51:57.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.868+0000 7f165ab67700 1 -- 172.21.15.123:0/543056101 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f164402ffb0 msgr2=0x7f1644032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:57.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.868+0000 7f165ab67700 1 --2- 172.21.15.123:0/543056101 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f164402ffb0 0x7f1644032470 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f16400096a0 tx=0x7f1640006b40).stop 2024-01-13T11:51:57.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.868+0000 7f165ab67700 1 -- 172.21.15.123:0/543056101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1654060220 msgr2=0x7f16541066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:51:57.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.868+0000 7f165ab67700 1 --2- 172.21.15.123:0/543056101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1654060220 0x7f16541066a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f1648013510 tx=0x7f164800b040).stop 2024-01-13T11:51:57.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.868+0000 7f165ab67700 1 -- 172.21.15.123:0/543056101 shutdown_connections 2024-01-13T11:51:57.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.868+0000 7f165ab67700 1 --2- 172.21.15.123:0/543056101 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f164402ffb0 0x7f1644032470 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:57.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.868+0000 7f165ab67700 1 --2- 172.21.15.123:0/543056101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1654060220 0x7f16541066a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:51:57.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.868+0000 7f165ab67700 1 -- 172.21.15.123:0/543056101 >> 172.21.15.123:0/543056101 conn(0x7f16540f0b70 msgr2=0x7f16540f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:51:57.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.869+0000 7f165ab67700 1 -- 172.21.15.123:0/543056101 shutdown_connections 2024-01-13T11:51:57.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:51:57.869+0000 7f165ab67700 1 -- 172.21.15.123:0/543056101 wait complete. 2024-01-13T11:51:57.859 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:51:58.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:58 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:51:57.868149+0000 mon.smithi059 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.123:0/543056101' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:51:59.245 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:51:59.246 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:51:59.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:51:59 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:57.520849+0000 mgr.smithi059.hmyldj (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:00.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.365+0000 7fa8c8849700 1 -- 172.21.15.123:0/4241495968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8c40f5410 msgr2=0x7fa8c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:00.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.365+0000 7fa8c8849700 1 --2- 172.21.15.123:0/4241495968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8c40f5410 0x7fa8c40f5830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fa8b8009fb0 tx=0x7fa8b8009b30).stop 2024-01-13T11:52:00.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.365+0000 7fa8c8849700 1 -- 172.21.15.123:0/4241495968 shutdown_connections 2024-01-13T11:52:00.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.365+0000 7fa8c8849700 1 --2- 172.21.15.123:0/4241495968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8c40f5410 0x7fa8c40f5830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:00.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.365+0000 7fa8c8849700 1 -- 172.21.15.123:0/4241495968 >> 172.21.15.123:0/4241495968 conn(0x7fa8c40f0b70 msgr2=0x7fa8c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:00.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.365+0000 7fa8c8849700 1 -- 172.21.15.123:0/4241495968 shutdown_connections 2024-01-13T11:52:00.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.365+0000 7fa8c8849700 1 -- 172.21.15.123:0/4241495968 wait complete. 2024-01-13T11:52:00.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.366+0000 7fa8c8849700 1 Processor -- start 2024-01-13T11:52:00.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.366+0000 7fa8c8849700 1 -- start start 2024-01-13T11:52:00.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.366+0000 7fa8c8849700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8c40f5410 0x7fa8c40fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:00.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.366+0000 7fa8c8849700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c40fb820 con 0x7fa8c40f5410 2024-01-13T11:52:00.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.366+0000 7fa8c259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8c40f5410 0x7fa8c40fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:00.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.366+0000 7fa8c259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8c40f5410 0x7fa8c40fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:55340/0 (socket says 172.21.15.123:55340) 2024-01-13T11:52:00.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.366+0000 7fa8c259c700 1 -- 172.21.15.123:0/478369975 learned_addr learned my addr 172.21.15.123:0/478369975 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:00.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.367+0000 7fa8c259c700 1 -- 172.21.15.123:0/478369975 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b8005040 con 0x7fa8c40f5410 2024-01-13T11:52:00.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.367+0000 7fa8c259c700 1 --2- 172.21.15.123:0/478369975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8c40f5410 0x7fa8c40fd3a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa8b8013510 tx=0x7fa8b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:00.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.367+0000 7fa8b2ffd700 1 -- 172.21.15.123:0/478369975 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8b8009320 con 0x7fa8c40f5410 2024-01-13T11:52:00.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.367+0000 7fa8c8849700 1 -- 172.21.15.123:0/478369975 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c40fb9c0 con 0x7fa8c40f5410 2024-01-13T11:52:00.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.367+0000 7fa8b2ffd700 1 -- 172.21.15.123:0/478369975 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8b8013f00 con 0x7fa8c40f5410 2024-01-13T11:52:00.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.367+0000 7fa8b2ffd700 1 -- 172.21.15.123:0/478369975 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8b8014d90 con 0x7fa8c40f5410 2024-01-13T11:52:00.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.367+0000 7fa8c8849700 1 -- 172.21.15.123:0/478369975 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c40fbe50 con 0x7fa8c40f5410 2024-01-13T11:52:00.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.368+0000 7fa8b2ffd700 1 -- 172.21.15.123:0/478369975 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8b80095f0 con 0x7fa8c40f5410 2024-01-13T11:52:00.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.368+0000 7fa8c8849700 1 -- 172.21.15.123:0/478369975 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c4059100 con 0x7fa8c40f5410 2024-01-13T11:52:00.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.368+0000 7fa8b2ffd700 1 --2- 172.21.15.123:0/478369975 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa8ac02ff60 0x7fa8ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:00.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.368+0000 7fa8b2ffd700 1 -- 172.21.15.123:0/478369975 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8b8041000 con 0x7fa8c40f5410 2024-01-13T11:52:00.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.368+0000 7fa8c1d9b700 1 --2- 172.21.15.123:0/478369975 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa8ac02ff60 0x7fa8ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:00.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.368+0000 7fa8c1d9b700 1 --2- 172.21.15.123:0/478369975 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa8ac02ff60 0x7fa8ac032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fa8b40096a0 tx=0x7fa8b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:00.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.373+0000 7fa8b2ffd700 1 -- 172.21.15.123:0/478369975 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8b8014420 con 0x7fa8c40f5410 2024-01-13T11:52:00.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.580+0000 7fa8c8849700 1 -- 172.21.15.123:0/478369975 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8c40fae60 con 0x7fa8c40f5410 2024-01-13T11:52:00.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.581+0000 7fa8b2ffd700 1 -- 172.21.15.123:0/478369975 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa8b8010230 con 0x7fa8c40f5410 2024-01-13T11:52:00.571 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:00.571 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:00.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.583+0000 7fa8c8849700 1 -- 172.21.15.123:0/478369975 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa8ac02ff60 msgr2=0x7fa8ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:00.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.583+0000 7fa8c8849700 1 --2- 172.21.15.123:0/478369975 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa8ac02ff60 0x7fa8ac032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fa8b40096a0 tx=0x7fa8b4006b40).stop 2024-01-13T11:52:00.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.583+0000 7fa8c8849700 1 -- 172.21.15.123:0/478369975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8c40f5410 msgr2=0x7fa8c40fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:00.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.583+0000 7fa8c8849700 1 --2- 172.21.15.123:0/478369975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8c40f5410 0x7fa8c40fd3a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa8b8013510 tx=0x7fa8b800b040).stop 2024-01-13T11:52:00.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.583+0000 7fa8c8849700 1 -- 172.21.15.123:0/478369975 shutdown_connections 2024-01-13T11:52:00.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.583+0000 7fa8c8849700 1 --2- 172.21.15.123:0/478369975 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa8ac02ff60 0x7fa8ac032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:00.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.583+0000 7fa8c8849700 1 --2- 172.21.15.123:0/478369975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8c40f5410 0x7fa8c40fd3a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:00.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.583+0000 7fa8c8849700 1 -- 172.21.15.123:0/478369975 >> 172.21.15.123:0/478369975 conn(0x7fa8c40f0b70 msgr2=0x7fa8c405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:00.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.583+0000 7fa8c8849700 1 -- 172.21.15.123:0/478369975 shutdown_connections 2024-01-13T11:52:00.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:00.583+0000 7fa8c8849700 1 -- 172.21.15.123:0/478369975 wait complete. 2024-01-13T11:52:00.573 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:01.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:01 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:51:59.521150+0000 mgr.smithi059.hmyldj (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:01.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:01 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:00.582608+0000 mon.smithi059 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.123:0/478369975' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:01.953 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:01.953 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:03.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.135+0000 7f8b726f0700 1 -- 172.21.15.123:0/2363677602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6c060280 msgr2=0x7f8b6c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.135+0000 7f8b726f0700 1 --2- 172.21.15.123:0/2363677602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6c060280 0x7f8b6c0f90f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f8b60009fb0 tx=0x7f8b60009b30).stop 2024-01-13T11:52:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.135+0000 7f8b726f0700 1 -- 172.21.15.123:0/2363677602 shutdown_connections 2024-01-13T11:52:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.135+0000 7f8b726f0700 1 --2- 172.21.15.123:0/2363677602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6c060280 0x7f8b6c0f90f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.135+0000 7f8b726f0700 1 -- 172.21.15.123:0/2363677602 >> 172.21.15.123:0/2363677602 conn(0x7f8b6c0f0b70 msgr2=0x7f8b6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.135+0000 7f8b726f0700 1 -- 172.21.15.123:0/2363677602 shutdown_connections 2024-01-13T11:52:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.135+0000 7f8b726f0700 1 -- 172.21.15.123:0/2363677602 wait complete. 2024-01-13T11:52:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.136+0000 7f8b726f0700 1 Processor -- start 2024-01-13T11:52:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.136+0000 7f8b726f0700 1 -- start start 2024-01-13T11:52:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.136+0000 7f8b726f0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6c060280 0x7f8b6c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:03.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.136+0000 7f8b726f0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c0f4ff0 con 0x7f8b6c060280 2024-01-13T11:52:03.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.137+0000 7f8b6bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6c060280 0x7f8b6c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:03.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.137+0000 7f8b6bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6c060280 0x7f8b6c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:55362/0 (socket says 172.21.15.123:55362) 2024-01-13T11:52:03.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.137+0000 7f8b6bfff700 1 -- 172.21.15.123:0/3774369054 learned_addr learned my addr 172.21.15.123:0/3774369054 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:03.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.137+0000 7f8b6bfff700 1 -- 172.21.15.123:0/3774369054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b60005040 con 0x7f8b6c060280 2024-01-13T11:52:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.137+0000 7f8b6bfff700 1 --2- 172.21.15.123:0/3774369054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6c060280 0x7f8b6c0f6b70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f8b60009830 tx=0x7f8b6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.137+0000 7f8b68ff9700 1 -- 172.21.15.123:0/3774369054 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b60009320 con 0x7f8b6c060280 2024-01-13T11:52:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.137+0000 7f8b68ff9700 1 -- 172.21.15.123:0/3774369054 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b600136c0 con 0x7f8b6c060280 2024-01-13T11:52:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.137+0000 7f8b726f0700 1 -- 172.21.15.123:0/3774369054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b6c0f5190 con 0x7f8b6c060280 2024-01-13T11:52:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.137+0000 7f8b68ff9700 1 -- 172.21.15.123:0/3774369054 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b6001ee90 con 0x7f8b6c060280 2024-01-13T11:52:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.137+0000 7f8b726f0700 1 -- 172.21.15.123:0/3774369054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b6c0f5620 con 0x7f8b6c060280 2024-01-13T11:52:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.138+0000 7f8b68ff9700 1 -- 172.21.15.123:0/3774369054 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8b6001e420 con 0x7f8b6c060280 2024-01-13T11:52:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.138+0000 7f8b726f0700 1 -- 172.21.15.123:0/3774369054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b6c059100 con 0x7f8b6c060280 2024-01-13T11:52:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.138+0000 7f8b68ff9700 1 --2- 172.21.15.123:0/3774369054 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8b5402ff10 0x7f8b540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.138+0000 7f8b68ff9700 1 -- 172.21.15.123:0/3774369054 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8b60040fb0 con 0x7f8b6c060280 2024-01-13T11:52:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.138+0000 7f8b6b7fe700 1 --2- 172.21.15.123:0/3774369054 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8b5402ff10 0x7f8b540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:03.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.139+0000 7f8b6b7fe700 1 --2- 172.21.15.123:0/3774369054 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8b5402ff10 0x7f8b540323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f8b5c0096a0 tx=0x7f8b5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:03.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.143+0000 7f8b68ff9700 1 -- 172.21.15.123:0/3774369054 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b6006d2b0 con 0x7f8b6c060280 2024-01-13T11:52:03.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.349+0000 7f8b726f0700 1 -- 172.21.15.123:0/3774369054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b6c0fbf50 con 0x7f8b6c060280 2024-01-13T11:52:03.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.350+0000 7f8b68ff9700 1 -- 172.21.15.123:0/3774369054 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8b6001c790 con 0x7f8b6c060280 2024-01-13T11:52:03.339 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:03.339 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:03.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.351+0000 7f8b726f0700 1 -- 172.21.15.123:0/3774369054 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8b5402ff10 msgr2=0x7f8b540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:03.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.351+0000 7f8b726f0700 1 --2- 172.21.15.123:0/3774369054 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8b5402ff10 0x7f8b540323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f8b5c0096a0 tx=0x7f8b5c006b40).stop 2024-01-13T11:52:03.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.351+0000 7f8b726f0700 1 -- 172.21.15.123:0/3774369054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6c060280 msgr2=0x7f8b6c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:03.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.351+0000 7f8b726f0700 1 --2- 172.21.15.123:0/3774369054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6c060280 0x7f8b6c0f6b70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f8b60009830 tx=0x7f8b6000b040).stop 2024-01-13T11:52:03.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.351+0000 7f8b726f0700 1 -- 172.21.15.123:0/3774369054 shutdown_connections 2024-01-13T11:52:03.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.352+0000 7f8b726f0700 1 --2- 172.21.15.123:0/3774369054 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8b5402ff10 0x7f8b540323d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:03.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.352+0000 7f8b726f0700 1 --2- 172.21.15.123:0/3774369054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6c060280 0x7f8b6c0f6b70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:03.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.352+0000 7f8b726f0700 1 -- 172.21.15.123:0/3774369054 >> 172.21.15.123:0/3774369054 conn(0x7f8b6c0f0b70 msgr2=0x7f8b6c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:03.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.352+0000 7f8b726f0700 1 -- 172.21.15.123:0/3774369054 shutdown_connections 2024-01-13T11:52:03.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:03.352+0000 7f8b726f0700 1 -- 172.21.15.123:0/3774369054 wait complete. 2024-01-13T11:52:03.342 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:03.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:03 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:01.521535+0000 mgr.smithi059.hmyldj (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:04.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:04 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:03.351221+0000 mon.smithi059 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.123:0/3774369054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:04.736 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:04.737 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:05.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:05 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:03.521848+0000 mgr.smithi059.hmyldj (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:05.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.955+0000 7f98f0ed3700 1 -- 172.21.15.123:0/2473642534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98ec060280 msgr2=0x7f98ec0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:05.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.955+0000 7f98f0ed3700 1 --2- 172.21.15.123:0/2473642534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98ec060280 0x7f98ec0f90f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f98dc009fb0 tx=0x7f98dc009b30).stop 2024-01-13T11:52:05.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.956+0000 7f98f0ed3700 1 -- 172.21.15.123:0/2473642534 shutdown_connections 2024-01-13T11:52:05.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.956+0000 7f98f0ed3700 1 --2- 172.21.15.123:0/2473642534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98ec060280 0x7f98ec0f90f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:05.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.956+0000 7f98f0ed3700 1 -- 172.21.15.123:0/2473642534 >> 172.21.15.123:0/2473642534 conn(0x7f98ec0f0b70 msgr2=0x7f98ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:05.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.956+0000 7f98f0ed3700 1 -- 172.21.15.123:0/2473642534 shutdown_connections 2024-01-13T11:52:05.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.956+0000 7f98f0ed3700 1 -- 172.21.15.123:0/2473642534 wait complete. 2024-01-13T11:52:05.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.956+0000 7f98f0ed3700 1 Processor -- start 2024-01-13T11:52:05.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.956+0000 7f98f0ed3700 1 -- start start 2024-01-13T11:52:05.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.956+0000 7f98f0ed3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98ec060280 0x7f98ec1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:05.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.956+0000 7f98f0ed3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec106bd0 con 0x7f98ec060280 2024-01-13T11:52:05.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.956+0000 7f98ea59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98ec060280 0x7f98ec1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:05.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.957+0000 7f98ea59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98ec060280 0x7f98ec1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:55384/0 (socket says 172.21.15.123:55384) 2024-01-13T11:52:05.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.957+0000 7f98ea59c700 1 -- 172.21.15.123:0/2978984579 learned_addr learned my addr 172.21.15.123:0/2978984579 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:05.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.957+0000 7f98ea59c700 1 -- 172.21.15.123:0/2978984579 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc005040 con 0x7f98ec060280 2024-01-13T11:52:05.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.957+0000 7f98ea59c700 1 --2- 172.21.15.123:0/2978984579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98ec060280 0x7f98ec1066e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f98dc013510 tx=0x7f98dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:05.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.957+0000 7f98daffd700 1 -- 172.21.15.123:0/2978984579 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98dc009320 con 0x7f98ec060280 2024-01-13T11:52:05.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.957+0000 7f98daffd700 1 -- 172.21.15.123:0/2978984579 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98dc013f00 con 0x7f98ec060280 2024-01-13T11:52:05.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.957+0000 7f98f0ed3700 1 -- 172.21.15.123:0/2978984579 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98ec106dd0 con 0x7f98ec060280 2024-01-13T11:52:05.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.958+0000 7f98daffd700 1 -- 172.21.15.123:0/2978984579 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98dc014dc0 con 0x7f98ec060280 2024-01-13T11:52:05.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.958+0000 7f98f0ed3700 1 -- 172.21.15.123:0/2978984579 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ec107260 con 0x7f98ec060280 2024-01-13T11:52:05.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.958+0000 7f98daffd700 1 -- 172.21.15.123:0/2978984579 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f98dc0263a0 con 0x7f98ec060280 2024-01-13T11:52:05.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.959+0000 7f98daffd700 1 --2- 172.21.15.123:0/2978984579 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f98d402ffb0 0x7f98d4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:05.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.959+0000 7f98f0ed3700 1 -- 172.21.15.123:0/2978984579 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98ec045ad0 con 0x7f98ec060280 2024-01-13T11:52:05.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.959+0000 7f98daffd700 1 -- 172.21.15.123:0/2978984579 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f98dc041090 con 0x7f98ec060280 2024-01-13T11:52:05.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.959+0000 7f98e9d9b700 1 --2- 172.21.15.123:0/2978984579 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f98d402ffb0 0x7f98d4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:05.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.959+0000 7f98e9d9b700 1 --2- 172.21.15.123:0/2978984579 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f98d402ffb0 0x7f98d4032470 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f98e00096a0 tx=0x7f98e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:05.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:05.964+0000 7f98daffd700 1 -- 172.21.15.123:0/2978984579 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98dc012920 con 0x7f98ec060280 2024-01-13T11:52:06.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:06.170+0000 7f98f0ed3700 1 -- 172.21.15.123:0/2978984579 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f98ec108110 con 0x7f98ec060280 2024-01-13T11:52:06.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:06.171+0000 7f98daffd700 1 -- 172.21.15.123:0/2978984579 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f98dc02b020 con 0x7f98ec060280 2024-01-13T11:52:06.160 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:06.160 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:06.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:06.172+0000 7f98f0ed3700 1 -- 172.21.15.123:0/2978984579 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f98d402ffb0 msgr2=0x7f98d4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:06.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:06.172+0000 7f98f0ed3700 1 --2- 172.21.15.123:0/2978984579 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f98d402ffb0 0x7f98d4032470 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f98e00096a0 tx=0x7f98e0006b40).stop 2024-01-13T11:52:06.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:06.172+0000 7f98f0ed3700 1 -- 172.21.15.123:0/2978984579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98ec060280 msgr2=0x7f98ec1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:06.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:06.172+0000 7f98f0ed3700 1 --2- 172.21.15.123:0/2978984579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98ec060280 0x7f98ec1066e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f98dc013510 tx=0x7f98dc00b040).stop 2024-01-13T11:52:06.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:06.172+0000 7f98f0ed3700 1 -- 172.21.15.123:0/2978984579 shutdown_connections 2024-01-13T11:52:06.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:06.172+0000 7f98f0ed3700 1 --2- 172.21.15.123:0/2978984579 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f98d402ffb0 0x7f98d4032470 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:06.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:06.172+0000 7f98f0ed3700 1 --2- 172.21.15.123:0/2978984579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98ec060280 0x7f98ec1066e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:06.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:06.172+0000 7f98f0ed3700 1 -- 172.21.15.123:0/2978984579 >> 172.21.15.123:0/2978984579 conn(0x7f98ec0f0b70 msgr2=0x7f98ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:06.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:06.172+0000 7f98f0ed3700 1 -- 172.21.15.123:0/2978984579 shutdown_connections 2024-01-13T11:52:06.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:06.172+0000 7f98f0ed3700 1 -- 172.21.15.123:0/2978984579 wait complete. 2024-01-13T11:52:06.163 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:06.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:06 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:06.172139+0000 mon.smithi059 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.123:0/2978984579' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:07.564 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:07.565 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:07.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:05.522202+0000 mgr.smithi059.hmyldj (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:08.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.705+0000 7f41c4b91700 1 -- 172.21.15.123:0/2141460227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41c00f5410 msgr2=0x7f41c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:08.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.705+0000 7f41c4b91700 1 --2- 172.21.15.123:0/2141460227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41c00f5410 0x7f41c00f5830 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f41a8009fb0 tx=0x7f41a8009b30).stop 2024-01-13T11:52:08.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.705+0000 7f41c4b91700 1 -- 172.21.15.123:0/2141460227 shutdown_connections 2024-01-13T11:52:08.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.705+0000 7f41c4b91700 1 --2- 172.21.15.123:0/2141460227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41c00f5410 0x7f41c00f5830 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:08.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.705+0000 7f41c4b91700 1 -- 172.21.15.123:0/2141460227 >> 172.21.15.123:0/2141460227 conn(0x7f41c00f0b70 msgr2=0x7f41c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:08.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.705+0000 7f41c4b91700 1 -- 172.21.15.123:0/2141460227 shutdown_connections 2024-01-13T11:52:08.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.705+0000 7f41c4b91700 1 -- 172.21.15.123:0/2141460227 wait complete. 2024-01-13T11:52:08.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.705+0000 7f41c4b91700 1 Processor -- start 2024-01-13T11:52:08.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.705+0000 7f41c4b91700 1 -- start start 2024-01-13T11:52:08.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.705+0000 7f41c4b91700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41c00f5410 0x7f41c00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:08.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.705+0000 7f41c4b91700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c00fb820 con 0x7f41c00f5410 2024-01-13T11:52:08.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.706+0000 7f41be59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41c00f5410 0x7f41c00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:08.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.706+0000 7f41be59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41c00f5410 0x7f41c00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:56572/0 (socket says 172.21.15.123:56572) 2024-01-13T11:52:08.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.706+0000 7f41be59c700 1 -- 172.21.15.123:0/2602126305 learned_addr learned my addr 172.21.15.123:0/2602126305 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:08.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.706+0000 7f41be59c700 1 -- 172.21.15.123:0/2602126305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a8005040 con 0x7f41c00f5410 2024-01-13T11:52:08.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.706+0000 7f41be59c700 1 --2- 172.21.15.123:0/2602126305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41c00f5410 0x7f41c00fd3a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f41a8014000 tx=0x7f41a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:08.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.706+0000 7f41b6ffd700 1 -- 172.21.15.123:0/2602126305 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41a80095f0 con 0x7f41c00f5410 2024-01-13T11:52:08.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.706+0000 7f41b6ffd700 1 -- 172.21.15.123:0/2602126305 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f41a8007e70 con 0x7f41c00f5410 2024-01-13T11:52:08.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.706+0000 7f41b6ffd700 1 -- 172.21.15.123:0/2602126305 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41a8014ea0 con 0x7f41c00f5410 2024-01-13T11:52:08.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.706+0000 7f41c4b91700 1 -- 172.21.15.123:0/2602126305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c00fb9c0 con 0x7f41c00f5410 2024-01-13T11:52:08.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.706+0000 7f41c4b91700 1 -- 172.21.15.123:0/2602126305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c00fbe50 con 0x7f41c00f5410 2024-01-13T11:52:08.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.707+0000 7f41b6ffd700 1 -- 172.21.15.123:0/2602126305 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41a8025070 con 0x7f41c00f5410 2024-01-13T11:52:08.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.707+0000 7f41b6ffd700 1 --2- 172.21.15.123:0/2602126305 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f41ac02ff60 0x7f41ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:08.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.707+0000 7f41b6ffd700 1 -- 172.21.15.123:0/2602126305 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f41a80409c0 con 0x7f41c00f5410 2024-01-13T11:52:08.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.707+0000 7f41c4b91700 1 -- 172.21.15.123:0/2602126305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c0059100 con 0x7f41c00f5410 2024-01-13T11:52:08.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.708+0000 7f41bdd9b700 1 --2- 172.21.15.123:0/2602126305 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f41ac02ff60 0x7f41ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:08.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.708+0000 7f41bdd9b700 1 --2- 172.21.15.123:0/2602126305 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f41ac02ff60 0x7f41ac032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f41b00096a0 tx=0x7f41b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:08.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.712+0000 7f41b6ffd700 1 -- 172.21.15.123:0/2602126305 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41a8026350 con 0x7f41c00f5410 2024-01-13T11:52:08.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.926+0000 7f41c4b91700 1 -- 172.21.15.123:0/2602126305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41c0045ad0 con 0x7f41c00f5410 2024-01-13T11:52:08.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.927+0000 7f41b6ffd700 1 -- 172.21.15.123:0/2602126305 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f41a8012330 con 0x7f41c00f5410 2024-01-13T11:52:08.916 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:08.916 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:08.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.929+0000 7f41c4b91700 1 -- 172.21.15.123:0/2602126305 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f41ac02ff60 msgr2=0x7f41ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:08.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.929+0000 7f41c4b91700 1 --2- 172.21.15.123:0/2602126305 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f41ac02ff60 0x7f41ac032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f41b00096a0 tx=0x7f41b0006b40).stop 2024-01-13T11:52:08.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.929+0000 7f41c4b91700 1 -- 172.21.15.123:0/2602126305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41c00f5410 msgr2=0x7f41c00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:08.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.929+0000 7f41c4b91700 1 --2- 172.21.15.123:0/2602126305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41c00f5410 0x7f41c00fd3a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f41a8014000 tx=0x7f41a800b040).stop 2024-01-13T11:52:08.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.929+0000 7f41c4b91700 1 -- 172.21.15.123:0/2602126305 shutdown_connections 2024-01-13T11:52:08.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.929+0000 7f41c4b91700 1 --2- 172.21.15.123:0/2602126305 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f41ac02ff60 0x7f41ac032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:08.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.929+0000 7f41c4b91700 1 --2- 172.21.15.123:0/2602126305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41c00f5410 0x7f41c00fd3a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:08.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.929+0000 7f41c4b91700 1 -- 172.21.15.123:0/2602126305 >> 172.21.15.123:0/2602126305 conn(0x7f41c00f0b70 msgr2=0x7f41c005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:08.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.929+0000 7f41c4b91700 1 -- 172.21.15.123:0/2602126305 shutdown_connections 2024-01-13T11:52:08.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:08.929+0000 7f41c4b91700 1 -- 172.21.15.123:0/2602126305 wait complete. 2024-01-13T11:52:08.919 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:09.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:09 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:07.522610+0000 mgr.smithi059.hmyldj (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:09.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:09 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:08.928273+0000 mon.smithi059 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.123:0/2602126305' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:10.322 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:10.322 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:11.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.497+0000 7fb3e73a0700 1 -- 172.21.15.123:0/981354015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e00f62a0 msgr2=0x7fb3e00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:11.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.497+0000 7fb3e73a0700 1 --2- 172.21.15.123:0/981354015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e00f62a0 0x7fb3e00f66c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fb3d0009fb0 tx=0x7fb3d0009b30).stop 2024-01-13T11:52:11.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.498+0000 7fb3e73a0700 1 -- 172.21.15.123:0/981354015 shutdown_connections 2024-01-13T11:52:11.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.498+0000 7fb3e73a0700 1 --2- 172.21.15.123:0/981354015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e00f62a0 0x7fb3e00f66c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:11.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.498+0000 7fb3e73a0700 1 -- 172.21.15.123:0/981354015 >> 172.21.15.123:0/981354015 conn(0x7fb3e00f1aa0 msgr2=0x7fb3e00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:11.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.498+0000 7fb3e73a0700 1 -- 172.21.15.123:0/981354015 shutdown_connections 2024-01-13T11:52:11.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.498+0000 7fb3e73a0700 1 -- 172.21.15.123:0/981354015 wait complete. 2024-01-13T11:52:11.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.498+0000 7fb3e73a0700 1 Processor -- start 2024-01-13T11:52:11.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.498+0000 7fb3e73a0700 1 -- start start 2024-01-13T11:52:11.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.498+0000 7fb3e73a0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e00f62a0 0x7fb3e01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:11.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.498+0000 7fb3e73a0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e01069a0 con 0x7fb3e00f62a0 2024-01-13T11:52:11.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.499+0000 7fb3e639e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e00f62a0 0x7fb3e01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:11.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.499+0000 7fb3e639e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e00f62a0 0x7fb3e01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:56598/0 (socket says 172.21.15.123:56598) 2024-01-13T11:52:11.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.499+0000 7fb3e639e700 1 -- 172.21.15.123:0/4012777814 learned_addr learned my addr 172.21.15.123:0/4012777814 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:11.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.499+0000 7fb3e639e700 1 -- 172.21.15.123:0/4012777814 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d0005040 con 0x7fb3e00f62a0 2024-01-13T11:52:11.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.499+0000 7fb3e639e700 1 --2- 172.21.15.123:0/4012777814 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e00f62a0 0x7fb3e01064b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fb3d0013510 tx=0x7fb3d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:11.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.499+0000 7fb3d6ffd700 1 -- 172.21.15.123:0/4012777814 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3d0009320 con 0x7fb3e00f62a0 2024-01-13T11:52:11.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.499+0000 7fb3d6ffd700 1 -- 172.21.15.123:0/4012777814 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3d0013f00 con 0x7fb3e00f62a0 2024-01-13T11:52:11.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.499+0000 7fb3d6ffd700 1 -- 172.21.15.123:0/4012777814 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3d0014dc0 con 0x7fb3e00f62a0 2024-01-13T11:52:11.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.499+0000 7fb3e73a0700 1 -- 172.21.15.123:0/4012777814 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e0106ba0 con 0x7fb3e00f62a0 2024-01-13T11:52:11.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.499+0000 7fb3e73a0700 1 -- 172.21.15.123:0/4012777814 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e0107030 con 0x7fb3e00f62a0 2024-01-13T11:52:11.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.500+0000 7fb3d6ffd700 1 -- 172.21.15.123:0/4012777814 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3d00095f0 con 0x7fb3e00f62a0 2024-01-13T11:52:11.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.500+0000 7fb3e73a0700 1 -- 172.21.15.123:0/4012777814 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e0045ad0 con 0x7fb3e00f62a0 2024-01-13T11:52:11.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.500+0000 7fb3d6ffd700 1 --2- 172.21.15.123:0/4012777814 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb3cc02ff60 0x7fb3cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:11.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.500+0000 7fb3d6ffd700 1 -- 172.21.15.123:0/4012777814 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3d0041100 con 0x7fb3e00f62a0 2024-01-13T11:52:11.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.500+0000 7fb3e5b9d700 1 --2- 172.21.15.123:0/4012777814 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb3cc02ff60 0x7fb3cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:11.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.501+0000 7fb3e5b9d700 1 --2- 172.21.15.123:0/4012777814 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb3cc02ff60 0x7fb3cc032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fb3dc0096a0 tx=0x7fb3dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:11.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.505+0000 7fb3d6ffd700 1 -- 172.21.15.123:0/4012777814 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3d0014420 con 0x7fb3e00f62a0 2024-01-13T11:52:11.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:11 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:09.522902+0000 mgr.smithi059.hmyldj (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:11.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.714+0000 7fb3e73a0700 1 -- 172.21.15.123:0/4012777814 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3e0107ee0 con 0x7fb3e00f62a0 2024-01-13T11:52:11.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.715+0000 7fb3d6ffd700 1 -- 172.21.15.123:0/4012777814 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb3d0010230 con 0x7fb3e00f62a0 2024-01-13T11:52:11.704 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:11.705 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:11.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.716+0000 7fb3e73a0700 1 -- 172.21.15.123:0/4012777814 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb3cc02ff60 msgr2=0x7fb3cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:11.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.716+0000 7fb3e73a0700 1 --2- 172.21.15.123:0/4012777814 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb3cc02ff60 0x7fb3cc032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fb3dc0096a0 tx=0x7fb3dc006b40).stop 2024-01-13T11:52:11.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.716+0000 7fb3e73a0700 1 -- 172.21.15.123:0/4012777814 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e00f62a0 msgr2=0x7fb3e01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:11.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.716+0000 7fb3e73a0700 1 --2- 172.21.15.123:0/4012777814 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e00f62a0 0x7fb3e01064b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fb3d0013510 tx=0x7fb3d000b040).stop 2024-01-13T11:52:11.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.717+0000 7fb3e73a0700 1 -- 172.21.15.123:0/4012777814 shutdown_connections 2024-01-13T11:52:11.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.717+0000 7fb3e73a0700 1 --2- 172.21.15.123:0/4012777814 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb3cc02ff60 0x7fb3cc032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:11.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.717+0000 7fb3e73a0700 1 --2- 172.21.15.123:0/4012777814 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3e00f62a0 0x7fb3e01064b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:11.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.717+0000 7fb3e73a0700 1 -- 172.21.15.123:0/4012777814 >> 172.21.15.123:0/4012777814 conn(0x7fb3e00f1aa0 msgr2=0x7fb3e00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:11.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.717+0000 7fb3e73a0700 1 -- 172.21.15.123:0/4012777814 shutdown_connections 2024-01-13T11:52:11.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:11.717+0000 7fb3e73a0700 1 -- 172.21.15.123:0/4012777814 wait complete. 2024-01-13T11:52:11.707 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:12.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:12 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:11.716327+0000 mon.smithi059 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.123:0/4012777814' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:13.090 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:13.091 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:13.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:13 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:11.523214+0000 mgr.smithi059.hmyldj (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:14.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.262+0000 7fe88b6dc700 1 -- 172.21.15.123:0/1203880327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8840f6510 msgr2=0x7fe8840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:14.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.262+0000 7fe88b6dc700 1 --2- 172.21.15.123:0/1203880327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8840f6510 0x7fe8840f6930 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fe874009fb0 tx=0x7fe874009b30).stop 2024-01-13T11:52:14.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.262+0000 7fe88b6dc700 1 -- 172.21.15.123:0/1203880327 shutdown_connections 2024-01-13T11:52:14.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.262+0000 7fe88b6dc700 1 --2- 172.21.15.123:0/1203880327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8840f6510 0x7fe8840f6930 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:14.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.262+0000 7fe88b6dc700 1 -- 172.21.15.123:0/1203880327 >> 172.21.15.123:0/1203880327 conn(0x7fe8840f1c70 msgr2=0x7fe8840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:14.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.262+0000 7fe88b6dc700 1 -- 172.21.15.123:0/1203880327 shutdown_connections 2024-01-13T11:52:14.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.262+0000 7fe88b6dc700 1 -- 172.21.15.123:0/1203880327 wait complete. 2024-01-13T11:52:14.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.263+0000 7fe88b6dc700 1 Processor -- start 2024-01-13T11:52:14.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.263+0000 7fe88b6dc700 1 -- start start 2024-01-13T11:52:14.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.263+0000 7fe88b6dc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8840f6510 0x7fe884106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:14.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.263+0000 7fe88b6dc700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe884106c20 con 0x7fe8840f6510 2024-01-13T11:52:14.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.264+0000 7fe889478700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8840f6510 0x7fe884106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:14.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.264+0000 7fe889478700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8840f6510 0x7fe884106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:56608/0 (socket says 172.21.15.123:56608) 2024-01-13T11:52:14.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.264+0000 7fe889478700 1 -- 172.21.15.123:0/3316203891 learned_addr learned my addr 172.21.15.123:0/3316203891 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:14.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.264+0000 7fe889478700 1 -- 172.21.15.123:0/3316203891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe874005040 con 0x7fe8840f6510 2024-01-13T11:52:14.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.264+0000 7fe889478700 1 --2- 172.21.15.123:0/3316203891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8840f6510 0x7fe884106730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fe874013510 tx=0x7fe87400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:14.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.264+0000 7fe879ffb700 1 -- 172.21.15.123:0/3316203891 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe874009320 con 0x7fe8840f6510 2024-01-13T11:52:14.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.264+0000 7fe88b6dc700 1 -- 172.21.15.123:0/3316203891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe884106e20 con 0x7fe8840f6510 2024-01-13T11:52:14.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.264+0000 7fe879ffb700 1 -- 172.21.15.123:0/3316203891 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe874013f00 con 0x7fe8840f6510 2024-01-13T11:52:14.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.264+0000 7fe879ffb700 1 -- 172.21.15.123:0/3316203891 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe874014d90 con 0x7fe8840f6510 2024-01-13T11:52:14.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.264+0000 7fe88b6dc700 1 -- 172.21.15.123:0/3316203891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8841072b0 con 0x7fe8840f6510 2024-01-13T11:52:14.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.265+0000 7fe88b6dc700 1 -- 172.21.15.123:0/3316203891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe884045ad0 con 0x7fe8840f6510 2024-01-13T11:52:14.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.265+0000 7fe879ffb700 1 -- 172.21.15.123:0/3316203891 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8740095f0 con 0x7fe8840f6510 2024-01-13T11:52:14.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.265+0000 7fe879ffb700 1 --2- 172.21.15.123:0/3316203891 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe87002ff60 0x7fe870032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:14.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.265+0000 7fe879ffb700 1 -- 172.21.15.123:0/3316203891 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe8740410c0 con 0x7fe8840f6510 2024-01-13T11:52:14.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.266+0000 7fe888c77700 1 --2- 172.21.15.123:0/3316203891 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe87002ff60 0x7fe870032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:14.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.266+0000 7fe888c77700 1 --2- 172.21.15.123:0/3316203891 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe87002ff60 0x7fe870032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fe8800096a0 tx=0x7fe880006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:14.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.270+0000 7fe879ffb700 1 -- 172.21.15.123:0/3316203891 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe87401d7f0 con 0x7fe8840f6510 2024-01-13T11:52:14.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.477+0000 7fe88b6dc700 1 -- 172.21.15.123:0/3316203891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8840fbf60 con 0x7fe8840f6510 2024-01-13T11:52:14.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.478+0000 7fe879ffb700 1 -- 172.21.15.123:0/3316203891 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe874010230 con 0x7fe8840f6510 2024-01-13T11:52:14.467 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:14.467 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:14.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.479+0000 7fe88b6dc700 1 -- 172.21.15.123:0/3316203891 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe87002ff60 msgr2=0x7fe870032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:14.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.479+0000 7fe88b6dc700 1 --2- 172.21.15.123:0/3316203891 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe87002ff60 0x7fe870032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fe8800096a0 tx=0x7fe880006b40).stop 2024-01-13T11:52:14.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.479+0000 7fe88b6dc700 1 -- 172.21.15.123:0/3316203891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8840f6510 msgr2=0x7fe884106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:14.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.479+0000 7fe88b6dc700 1 --2- 172.21.15.123:0/3316203891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8840f6510 0x7fe884106730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fe874013510 tx=0x7fe87400b040).stop 2024-01-13T11:52:14.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.479+0000 7fe88b6dc700 1 -- 172.21.15.123:0/3316203891 shutdown_connections 2024-01-13T11:52:14.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.479+0000 7fe88b6dc700 1 --2- 172.21.15.123:0/3316203891 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe87002ff60 0x7fe870032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:14.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.479+0000 7fe88b6dc700 1 --2- 172.21.15.123:0/3316203891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8840f6510 0x7fe884106730 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:14.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.479+0000 7fe88b6dc700 1 -- 172.21.15.123:0/3316203891 >> 172.21.15.123:0/3316203891 conn(0x7fe8840f1c70 msgr2=0x7fe8840f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:14.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.479+0000 7fe88b6dc700 1 -- 172.21.15.123:0/3316203891 shutdown_connections 2024-01-13T11:52:14.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:14.479+0000 7fe88b6dc700 1 -- 172.21.15.123:0/3316203891 wait complete. 2024-01-13T11:52:14.469 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:15.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:15 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:13.523574+0000 mgr.smithi059.hmyldj (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:15.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:15 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:14.478993+0000 mon.smithi059 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.123:0/3316203891' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:15.866 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:15.866 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:17.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.012+0000 7f9562146700 1 -- 172.21.15.123:0/3455487026 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f955c0f51a0 msgr2=0x7f955c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:17.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.012+0000 7f9562146700 1 --2- 172.21.15.123:0/3455487026 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f955c0f51a0 0x7f955c0f55c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f954c009fb0 tx=0x7f954c009b30).stop 2024-01-13T11:52:17.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.013+0000 7f9562146700 1 -- 172.21.15.123:0/3455487026 shutdown_connections 2024-01-13T11:52:17.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.013+0000 7f9562146700 1 --2- 172.21.15.123:0/3455487026 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f955c0f51a0 0x7f955c0f55c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:17.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.013+0000 7f9562146700 1 -- 172.21.15.123:0/3455487026 >> 172.21.15.123:0/3455487026 conn(0x7f955c0f09a0 msgr2=0x7f955c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:17.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.013+0000 7f9562146700 1 -- 172.21.15.123:0/3455487026 shutdown_connections 2024-01-13T11:52:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.013+0000 7f9562146700 1 -- 172.21.15.123:0/3455487026 wait complete. 2024-01-13T11:52:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.013+0000 7f9562146700 1 Processor -- start 2024-01-13T11:52:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.013+0000 7f9562146700 1 -- start start 2024-01-13T11:52:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.013+0000 7f9562146700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f955c0f51a0 0x7f955c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.013+0000 7f9562146700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955c060240 con 0x7f955c0f51a0 2024-01-13T11:52:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.014+0000 7f9561144700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f955c0f51a0 0x7f955c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.014+0000 7f9561144700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f955c0f51a0 0x7f955c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:56870/0 (socket says 172.21.15.123:56870) 2024-01-13T11:52:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.014+0000 7f9561144700 1 -- 172.21.15.123:0/2239852615 learned_addr learned my addr 172.21.15.123:0/2239852615 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.014+0000 7f9561144700 1 -- 172.21.15.123:0/2239852615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954c005040 con 0x7f955c0f51a0 2024-01-13T11:52:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.014+0000 7f9561144700 1 --2- 172.21.15.123:0/2239852615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f955c0f51a0 0x7f955c1091b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f954c013510 tx=0x7f954c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.014+0000 7f9551ffb700 1 -- 172.21.15.123:0/2239852615 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f954c009320 con 0x7f955c0f51a0 2024-01-13T11:52:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.014+0000 7f9551ffb700 1 -- 172.21.15.123:0/2239852615 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f954c013f00 con 0x7f955c0f51a0 2024-01-13T11:52:17.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.014+0000 7f9562146700 1 -- 172.21.15.123:0/2239852615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f955c060440 con 0x7f955c0f51a0 2024-01-13T11:52:17.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.014+0000 7f9551ffb700 1 -- 172.21.15.123:0/2239852615 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f954c014dc0 con 0x7f955c0f51a0 2024-01-13T11:52:17.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.014+0000 7f9562146700 1 -- 172.21.15.123:0/2239852615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f955c109860 con 0x7f955c0f51a0 2024-01-13T11:52:17.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.015+0000 7f9551ffb700 1 -- 172.21.15.123:0/2239852615 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f954c014420 con 0x7f955c0f51a0 2024-01-13T11:52:17.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.015+0000 7f9562146700 1 -- 172.21.15.123:0/2239852615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f955c045ad0 con 0x7f955c0f51a0 2024-01-13T11:52:17.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.015+0000 7f9551ffb700 1 --2- 172.21.15.123:0/2239852615 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f954802ff60 0x7f9548032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.015+0000 7f9551ffb700 1 -- 172.21.15.123:0/2239852615 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f954c041030 con 0x7f955c0f51a0 2024-01-13T11:52:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.016+0000 7f9560943700 1 --2- 172.21.15.123:0/2239852615 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f954802ff60 0x7f9548032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:17.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.016+0000 7f9560943700 1 --2- 172.21.15.123:0/2239852615 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f954802ff60 0x7f9548032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f95580096a0 tx=0x7f9558006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:17.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.020+0000 7f9551ffb700 1 -- 172.21.15.123:0/2239852615 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f954c0095f0 con 0x7f955c0f51a0 2024-01-13T11:52:17.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.226+0000 7f9562146700 1 -- 172.21.15.123:0/2239852615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f955c10a080 con 0x7f955c0f51a0 2024-01-13T11:52:17.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.227+0000 7f9551ffb700 1 -- 172.21.15.123:0/2239852615 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f954c01cb10 con 0x7f955c0f51a0 2024-01-13T11:52:17.216 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:17.216 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:17.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.229+0000 7f9562146700 1 -- 172.21.15.123:0/2239852615 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f954802ff60 msgr2=0x7f9548032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:17.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.229+0000 7f9562146700 1 --2- 172.21.15.123:0/2239852615 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f954802ff60 0x7f9548032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f95580096a0 tx=0x7f9558006b40).stop 2024-01-13T11:52:17.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.229+0000 7f9562146700 1 -- 172.21.15.123:0/2239852615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f955c0f51a0 msgr2=0x7f955c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:17.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.229+0000 7f9562146700 1 --2- 172.21.15.123:0/2239852615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f955c0f51a0 0x7f955c1091b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f954c013510 tx=0x7f954c00b040).stop 2024-01-13T11:52:17.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.229+0000 7f9562146700 1 -- 172.21.15.123:0/2239852615 shutdown_connections 2024-01-13T11:52:17.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.229+0000 7f9562146700 1 --2- 172.21.15.123:0/2239852615 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f954802ff60 0x7f9548032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:17.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.229+0000 7f9562146700 1 --2- 172.21.15.123:0/2239852615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f955c0f51a0 0x7f955c1091b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:17.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.229+0000 7f9562146700 1 -- 172.21.15.123:0/2239852615 >> 172.21.15.123:0/2239852615 conn(0x7f955c0f09a0 msgr2=0x7f955c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:17.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.229+0000 7f9562146700 1 -- 172.21.15.123:0/2239852615 shutdown_connections 2024-01-13T11:52:17.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:17.229+0000 7f9562146700 1 -- 172.21.15.123:0/2239852615 wait complete. 2024-01-13T11:52:17.219 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:17.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:17 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:15.523917+0000 mgr.smithi059.hmyldj (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:18.601 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:18.601 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:18.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:18 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:17.228206+0000 mon.smithi059 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.123:0/2239852615' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:19.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:19 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:17.524122+0000 mgr.smithi059.hmyldj (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:19.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.720+0000 7f0c97fff700 1 -- 172.21.15.123:0/632746659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c980f51a0 msgr2=0x7f0c980f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:19.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.720+0000 7f0c97fff700 1 --2- 172.21.15.123:0/632746659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c980f51a0 0x7f0c980f55c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f0c8c009fb0 tx=0x7f0c8c009b30).stop 2024-01-13T11:52:19.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.720+0000 7f0c97fff700 1 -- 172.21.15.123:0/632746659 shutdown_connections 2024-01-13T11:52:19.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.720+0000 7f0c97fff700 1 --2- 172.21.15.123:0/632746659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c980f51a0 0x7f0c980f55c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:19.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.720+0000 7f0c97fff700 1 -- 172.21.15.123:0/632746659 >> 172.21.15.123:0/632746659 conn(0x7f0c980f09a0 msgr2=0x7f0c980f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:19.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.720+0000 7f0c97fff700 1 -- 172.21.15.123:0/632746659 shutdown_connections 2024-01-13T11:52:19.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.720+0000 7f0c97fff700 1 -- 172.21.15.123:0/632746659 wait complete. 2024-01-13T11:52:19.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.721+0000 7f0c97fff700 1 Processor -- start 2024-01-13T11:52:19.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.721+0000 7f0c97fff700 1 -- start start 2024-01-13T11:52:19.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.721+0000 7f0c97fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c980f51a0 0x7f0c981053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:19.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.721+0000 7f0c97fff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c981058a0 con 0x7f0c980f51a0 2024-01-13T11:52:19.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.721+0000 7f0c96ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c980f51a0 0x7f0c981053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:19.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.721+0000 7f0c96ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c980f51a0 0x7f0c981053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:56880/0 (socket says 172.21.15.123:56880) 2024-01-13T11:52:19.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.721+0000 7f0c96ffd700 1 -- 172.21.15.123:0/732459272 learned_addr learned my addr 172.21.15.123:0/732459272 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:19.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.721+0000 7f0c96ffd700 1 -- 172.21.15.123:0/732459272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8c005040 con 0x7f0c980f51a0 2024-01-13T11:52:19.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.722+0000 7f0c96ffd700 1 --2- 172.21.15.123:0/732459272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c980f51a0 0x7f0c981053b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f0c8c013510 tx=0x7f0c8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:19.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.722+0000 7f0c7f7fe700 1 -- 172.21.15.123:0/732459272 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c8c009320 con 0x7f0c980f51a0 2024-01-13T11:52:19.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.722+0000 7f0c97fff700 1 -- 172.21.15.123:0/732459272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c98105aa0 con 0x7f0c980f51a0 2024-01-13T11:52:19.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.722+0000 7f0c7f7fe700 1 -- 172.21.15.123:0/732459272 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c8c013f00 con 0x7f0c980f51a0 2024-01-13T11:52:19.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.722+0000 7f0c7f7fe700 1 -- 172.21.15.123:0/732459272 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c8c014dc0 con 0x7f0c980f51a0 2024-01-13T11:52:19.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.722+0000 7f0c97fff700 1 -- 172.21.15.123:0/732459272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c98105f30 con 0x7f0c980f51a0 2024-01-13T11:52:19.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.723+0000 7f0c7f7fe700 1 -- 172.21.15.123:0/732459272 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c8c014420 con 0x7f0c980f51a0 2024-01-13T11:52:19.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.723+0000 7f0c97fff700 1 -- 172.21.15.123:0/732459272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c98045ad0 con 0x7f0c980f51a0 2024-01-13T11:52:19.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.723+0000 7f0c7f7fe700 1 --2- 172.21.15.123:0/732459272 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0c8002ff60 0x7f0c80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:19.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.723+0000 7f0c7f7fe700 1 -- 172.21.15.123:0/732459272 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c8c041020 con 0x7f0c980f51a0 2024-01-13T11:52:19.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.723+0000 7f0c967fc700 1 --2- 172.21.15.123:0/732459272 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0c8002ff60 0x7f0c80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:19.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.723+0000 7f0c967fc700 1 --2- 172.21.15.123:0/732459272 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0c8002ff60 0x7f0c80032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f0c880096a0 tx=0x7f0c88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:19.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.727+0000 7f0c7f7fe700 1 -- 172.21.15.123:0/732459272 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c8c0095f0 con 0x7f0c980f51a0 2024-01-13T11:52:19.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.933+0000 7f0c97fff700 1 -- 172.21.15.123:0/732459272 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c98106de0 con 0x7f0c980f51a0 2024-01-13T11:52:19.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.933+0000 7f0c7f7fe700 1 -- 172.21.15.123:0/732459272 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0c8c01cb10 con 0x7f0c980f51a0 2024-01-13T11:52:19.923 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:19.924 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:19.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.935+0000 7f0c97fff700 1 -- 172.21.15.123:0/732459272 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0c8002ff60 msgr2=0x7f0c80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:19.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.935+0000 7f0c97fff700 1 --2- 172.21.15.123:0/732459272 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0c8002ff60 0x7f0c80032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f0c880096a0 tx=0x7f0c88006b40).stop 2024-01-13T11:52:19.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.935+0000 7f0c97fff700 1 -- 172.21.15.123:0/732459272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c980f51a0 msgr2=0x7f0c981053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:19.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.935+0000 7f0c97fff700 1 --2- 172.21.15.123:0/732459272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c980f51a0 0x7f0c981053b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f0c8c013510 tx=0x7f0c8c00b040).stop 2024-01-13T11:52:19.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.935+0000 7f0c97fff700 1 -- 172.21.15.123:0/732459272 shutdown_connections 2024-01-13T11:52:19.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.935+0000 7f0c97fff700 1 --2- 172.21.15.123:0/732459272 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0c8002ff60 0x7f0c80032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:19.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.935+0000 7f0c97fff700 1 --2- 172.21.15.123:0/732459272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c980f51a0 0x7f0c981053b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:19.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.935+0000 7f0c97fff700 1 -- 172.21.15.123:0/732459272 >> 172.21.15.123:0/732459272 conn(0x7f0c980f09a0 msgr2=0x7f0c980f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:19.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.935+0000 7f0c97fff700 1 -- 172.21.15.123:0/732459272 shutdown_connections 2024-01-13T11:52:19.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:19.935+0000 7f0c97fff700 1 -- 172.21.15.123:0/732459272 wait complete. 2024-01-13T11:52:19.927 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:20.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:20 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:19.934793+0000 mon.smithi059 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.123:0/732459272' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:21.307 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:21.308 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:21.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:21 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:19.524562+0000 mgr.smithi059.hmyldj (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:22.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.427+0000 7fe939d3b700 1 -- 172.21.15.123:0/1653066945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9340f5410 msgr2=0x7fe9340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:22.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.427+0000 7fe939d3b700 1 --2- 172.21.15.123:0/1653066945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9340f5410 0x7fe9340f5830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fe91c009fb0 tx=0x7fe91c009b30).stop 2024-01-13T11:52:22.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.427+0000 7fe939d3b700 1 -- 172.21.15.123:0/1653066945 shutdown_connections 2024-01-13T11:52:22.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.427+0000 7fe939d3b700 1 --2- 172.21.15.123:0/1653066945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9340f5410 0x7fe9340f5830 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:22.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.427+0000 7fe939d3b700 1 -- 172.21.15.123:0/1653066945 >> 172.21.15.123:0/1653066945 conn(0x7fe9340f0b70 msgr2=0x7fe9340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:22.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.427+0000 7fe939d3b700 1 -- 172.21.15.123:0/1653066945 shutdown_connections 2024-01-13T11:52:22.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.427+0000 7fe939d3b700 1 -- 172.21.15.123:0/1653066945 wait complete. 2024-01-13T11:52:22.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.428+0000 7fe939d3b700 1 Processor -- start 2024-01-13T11:52:22.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.428+0000 7fe939d3b700 1 -- start start 2024-01-13T11:52:22.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.428+0000 7fe939d3b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9340f5410 0x7fe934106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:22.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.428+0000 7fe939d3b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe934106c10 con 0x7fe9340f5410 2024-01-13T11:52:22.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.428+0000 7fe9337fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9340f5410 0x7fe934106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:22.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.428+0000 7fe9337fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9340f5410 0x7fe934106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:56906/0 (socket says 172.21.15.123:56906) 2024-01-13T11:52:22.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.428+0000 7fe9337fe700 1 -- 172.21.15.123:0/113929041 learned_addr learned my addr 172.21.15.123:0/113929041 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:22.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.429+0000 7fe9337fe700 1 -- 172.21.15.123:0/113929041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c005040 con 0x7fe9340f5410 2024-01-13T11:52:22.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.429+0000 7fe9337fe700 1 --2- 172.21.15.123:0/113929041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9340f5410 0x7fe934106720 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fe91c013510 tx=0x7fe91c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:22.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.429+0000 7fe92bfff700 1 -- 172.21.15.123:0/113929041 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe91c009320 con 0x7fe9340f5410 2024-01-13T11:52:22.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.429+0000 7fe939d3b700 1 -- 172.21.15.123:0/113929041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe934106e10 con 0x7fe9340f5410 2024-01-13T11:52:22.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.429+0000 7fe939d3b700 1 -- 172.21.15.123:0/113929041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9341072a0 con 0x7fe9340f5410 2024-01-13T11:52:22.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.429+0000 7fe92bfff700 1 -- 172.21.15.123:0/113929041 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe91c013f00 con 0x7fe9340f5410 2024-01-13T11:52:22.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.429+0000 7fe92bfff700 1 -- 172.21.15.123:0/113929041 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe91c014d90 con 0x7fe9340f5410 2024-01-13T11:52:22.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.430+0000 7fe92bfff700 1 -- 172.21.15.123:0/113929041 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe91c014420 con 0x7fe9340f5410 2024-01-13T11:52:22.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.430+0000 7fe939d3b700 1 -- 172.21.15.123:0/113929041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe934045ad0 con 0x7fe9340f5410 2024-01-13T11:52:22.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.430+0000 7fe92bfff700 1 --2- 172.21.15.123:0/113929041 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe92002ff60 0x7fe920032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:22.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.430+0000 7fe92bfff700 1 -- 172.21.15.123:0/113929041 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe91c041040 con 0x7fe9340f5410 2024-01-13T11:52:22.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.430+0000 7fe932ffd700 1 --2- 172.21.15.123:0/113929041 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe92002ff60 0x7fe920032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:22.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.431+0000 7fe932ffd700 1 --2- 172.21.15.123:0/113929041 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe92002ff60 0x7fe920032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fe9240096a0 tx=0x7fe924006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:22.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.435+0000 7fe92bfff700 1 -- 172.21.15.123:0/113929041 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe91c0095f0 con 0x7fe9340f5410 2024-01-13T11:52:22.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.641+0000 7fe939d3b700 1 -- 172.21.15.123:0/113929041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9341080f0 con 0x7fe9340f5410 2024-01-13T11:52:22.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.642+0000 7fe92bfff700 1 -- 172.21.15.123:0/113929041 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe91c01db70 con 0x7fe9340f5410 2024-01-13T11:52:22.631 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:22.631 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:22.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.644+0000 7fe939d3b700 1 -- 172.21.15.123:0/113929041 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe92002ff60 msgr2=0x7fe920032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:22.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.644+0000 7fe939d3b700 1 --2- 172.21.15.123:0/113929041 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe92002ff60 0x7fe920032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fe9240096a0 tx=0x7fe924006b40).stop 2024-01-13T11:52:22.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.644+0000 7fe939d3b700 1 -- 172.21.15.123:0/113929041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9340f5410 msgr2=0x7fe934106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:22.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.644+0000 7fe939d3b700 1 --2- 172.21.15.123:0/113929041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9340f5410 0x7fe934106720 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fe91c013510 tx=0x7fe91c00b040).stop 2024-01-13T11:52:22.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.644+0000 7fe939d3b700 1 -- 172.21.15.123:0/113929041 shutdown_connections 2024-01-13T11:52:22.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.644+0000 7fe939d3b700 1 --2- 172.21.15.123:0/113929041 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe92002ff60 0x7fe920032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:22.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.644+0000 7fe939d3b700 1 --2- 172.21.15.123:0/113929041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9340f5410 0x7fe934106720 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:22.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.644+0000 7fe939d3b700 1 -- 172.21.15.123:0/113929041 >> 172.21.15.123:0/113929041 conn(0x7fe9340f0b70 msgr2=0x7fe93405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:22.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.644+0000 7fe939d3b700 1 -- 172.21.15.123:0/113929041 shutdown_connections 2024-01-13T11:52:22.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:22.644+0000 7fe939d3b700 1 -- 172.21.15.123:0/113929041 wait complete. 2024-01-13T11:52:22.634 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:23.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:23 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:21.524857+0000 mgr.smithi059.hmyldj (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:23.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:23 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:22.643256+0000 mon.smithi059 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.123:0/113929041' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:24.017 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:24.017 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:25.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.161+0000 7f4afac07700 1 -- 172.21.15.123:0/2195348705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4af40f6510 msgr2=0x7f4af40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:25.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.161+0000 7f4afac07700 1 --2- 172.21.15.123:0/2195348705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4af40f6510 0x7f4af40f6930 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f4ae8009fb0 tx=0x7f4ae8009b30).stop 2024-01-13T11:52:25.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.161+0000 7f4afac07700 1 -- 172.21.15.123:0/2195348705 shutdown_connections 2024-01-13T11:52:25.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.161+0000 7f4afac07700 1 --2- 172.21.15.123:0/2195348705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4af40f6510 0x7f4af40f6930 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:25.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.161+0000 7f4afac07700 1 -- 172.21.15.123:0/2195348705 >> 172.21.15.123:0/2195348705 conn(0x7f4af40f1c70 msgr2=0x7f4af40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:25.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.161+0000 7f4afac07700 1 -- 172.21.15.123:0/2195348705 shutdown_connections 2024-01-13T11:52:25.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.161+0000 7f4afac07700 1 -- 172.21.15.123:0/2195348705 wait complete. 2024-01-13T11:52:25.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.161+0000 7f4afac07700 1 Processor -- start 2024-01-13T11:52:25.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.161+0000 7f4afac07700 1 -- start start 2024-01-13T11:52:25.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.161+0000 7f4afac07700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4af40f6510 0x7f4af4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:25.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.161+0000 7f4afac07700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af4106c20 con 0x7f4af40f6510 2024-01-13T11:52:25.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.162+0000 7f4af89a3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4af40f6510 0x7f4af4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:25.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.162+0000 7f4af89a3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4af40f6510 0x7f4af4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:56926/0 (socket says 172.21.15.123:56926) 2024-01-13T11:52:25.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.162+0000 7f4af89a3700 1 -- 172.21.15.123:0/2658357386 learned_addr learned my addr 172.21.15.123:0/2658357386 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:25.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.162+0000 7f4af89a3700 1 -- 172.21.15.123:0/2658357386 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae8005040 con 0x7f4af40f6510 2024-01-13T11:52:25.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.162+0000 7f4af89a3700 1 --2- 172.21.15.123:0/2658357386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4af40f6510 0x7f4af4106730 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f4ae8013510 tx=0x7f4ae800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:25.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.162+0000 7f4af17fa700 1 -- 172.21.15.123:0/2658357386 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ae8009320 con 0x7f4af40f6510 2024-01-13T11:52:25.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.162+0000 7f4af17fa700 1 -- 172.21.15.123:0/2658357386 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ae8013f00 con 0x7f4af40f6510 2024-01-13T11:52:25.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.162+0000 7f4afac07700 1 -- 172.21.15.123:0/2658357386 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af4106e20 con 0x7f4af40f6510 2024-01-13T11:52:25.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.162+0000 7f4af17fa700 1 -- 172.21.15.123:0/2658357386 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ae8014d90 con 0x7f4af40f6510 2024-01-13T11:52:25.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.162+0000 7f4afac07700 1 -- 172.21.15.123:0/2658357386 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af41072b0 con 0x7f4af40f6510 2024-01-13T11:52:25.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.163+0000 7f4af17fa700 1 -- 172.21.15.123:0/2658357386 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ae80095f0 con 0x7f4af40f6510 2024-01-13T11:52:25.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.163+0000 7f4afac07700 1 -- 172.21.15.123:0/2658357386 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af4045ad0 con 0x7f4af40f6510 2024-01-13T11:52:25.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.163+0000 7f4af17fa700 1 --2- 172.21.15.123:0/2658357386 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f4ae402ff60 0x7f4ae4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:25.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.163+0000 7f4af17fa700 1 -- 172.21.15.123:0/2658357386 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ae8041000 con 0x7f4af40f6510 2024-01-13T11:52:25.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.163+0000 7f4af3fff700 1 --2- 172.21.15.123:0/2658357386 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f4ae402ff60 0x7f4ae4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:25.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.164+0000 7f4af3fff700 1 --2- 172.21.15.123:0/2658357386 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f4ae402ff60 0x7f4ae4032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f4ae00096a0 tx=0x7f4ae0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:25.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.168+0000 7f4af17fa700 1 -- 172.21.15.123:0/2658357386 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ae8014420 con 0x7f4af40f6510 2024-01-13T11:52:25.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.375+0000 7f4afac07700 1 -- 172.21.15.123:0/2658357386 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4af40fbf60 con 0x7f4af40f6510 2024-01-13T11:52:25.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.377+0000 7f4af17fa700 1 -- 172.21.15.123:0/2658357386 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4ae801d7f0 con 0x7f4af40f6510 2024-01-13T11:52:25.365 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:25.365 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:25.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.377+0000 7f4afac07700 1 -- 172.21.15.123:0/2658357386 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f4ae402ff60 msgr2=0x7f4ae4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:25.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.377+0000 7f4afac07700 1 --2- 172.21.15.123:0/2658357386 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f4ae402ff60 0x7f4ae4032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f4ae00096a0 tx=0x7f4ae0006b40).stop 2024-01-13T11:52:25.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.377+0000 7f4afac07700 1 -- 172.21.15.123:0/2658357386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4af40f6510 msgr2=0x7f4af4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:25.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.377+0000 7f4afac07700 1 --2- 172.21.15.123:0/2658357386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4af40f6510 0x7f4af4106730 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f4ae8013510 tx=0x7f4ae800b040).stop 2024-01-13T11:52:25.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.378+0000 7f4afac07700 1 -- 172.21.15.123:0/2658357386 shutdown_connections 2024-01-13T11:52:25.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.378+0000 7f4afac07700 1 --2- 172.21.15.123:0/2658357386 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f4ae402ff60 0x7f4ae4032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:25.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.378+0000 7f4afac07700 1 --2- 172.21.15.123:0/2658357386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4af40f6510 0x7f4af4106730 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:25.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.378+0000 7f4afac07700 1 -- 172.21.15.123:0/2658357386 >> 172.21.15.123:0/2658357386 conn(0x7f4af40f1c70 msgr2=0x7f4af40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:25.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.378+0000 7f4afac07700 1 -- 172.21.15.123:0/2658357386 shutdown_connections 2024-01-13T11:52:25.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:25.378+0000 7f4afac07700 1 -- 172.21.15.123:0/2658357386 wait complete. 2024-01-13T11:52:25.368 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:25.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:25 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:23.525183+0000 mgr.smithi059.hmyldj (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:26.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:26 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:25.377067+0000 mon.smithi059 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.123:0/2658357386' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:26.760 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:26.760 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:27.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:27 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:25.525505+0000 mgr.smithi059.hmyldj (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:27.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.904+0000 7f7b711ab700 1 -- 172.21.15.123:0/2285779716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b6c0f5410 msgr2=0x7f7b6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:27.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.904+0000 7f7b711ab700 1 --2- 172.21.15.123:0/2285779716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b6c0f5410 0x7f7b6c0f5830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f7b54009fb0 tx=0x7f7b54009b30).stop 2024-01-13T11:52:27.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.905+0000 7f7b711ab700 1 -- 172.21.15.123:0/2285779716 shutdown_connections 2024-01-13T11:52:27.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.905+0000 7f7b711ab700 1 --2- 172.21.15.123:0/2285779716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b6c0f5410 0x7f7b6c0f5830 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:27.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.905+0000 7f7b711ab700 1 -- 172.21.15.123:0/2285779716 >> 172.21.15.123:0/2285779716 conn(0x7f7b6c0f0b70 msgr2=0x7f7b6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:27.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.905+0000 7f7b711ab700 1 -- 172.21.15.123:0/2285779716 shutdown_connections 2024-01-13T11:52:27.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.905+0000 7f7b711ab700 1 -- 172.21.15.123:0/2285779716 wait complete. 2024-01-13T11:52:27.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.905+0000 7f7b711ab700 1 Processor -- start 2024-01-13T11:52:27.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.905+0000 7f7b711ab700 1 -- start start 2024-01-13T11:52:27.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.905+0000 7f7b711ab700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b6c0f5410 0x7f7b6c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:27.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.905+0000 7f7b711ab700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c105b20 con 0x7f7b6c0f5410 2024-01-13T11:52:27.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.906+0000 7f7b6ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b6c0f5410 0x7f7b6c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:27.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.906+0000 7f7b6ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b6c0f5410 0x7f7b6c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:34820/0 (socket says 172.21.15.123:34820) 2024-01-13T11:52:27.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.906+0000 7f7b6ad9d700 1 -- 172.21.15.123:0/2617778979 learned_addr learned my addr 172.21.15.123:0/2617778979 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:27.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.906+0000 7f7b6ad9d700 1 -- 172.21.15.123:0/2617778979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b54005040 con 0x7f7b6c0f5410 2024-01-13T11:52:27.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.906+0000 7f7b6ad9d700 1 --2- 172.21.15.123:0/2617778979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b6c0f5410 0x7f7b6c105630 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f7b54013510 tx=0x7f7b5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:27.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.906+0000 7f7b637fe700 1 -- 172.21.15.123:0/2617778979 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b54009320 con 0x7f7b6c0f5410 2024-01-13T11:52:27.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.906+0000 7f7b711ab700 1 -- 172.21.15.123:0/2617778979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6c105d20 con 0x7f7b6c0f5410 2024-01-13T11:52:27.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.906+0000 7f7b637fe700 1 -- 172.21.15.123:0/2617778979 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b54013f00 con 0x7f7b6c0f5410 2024-01-13T11:52:27.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.906+0000 7f7b637fe700 1 -- 172.21.15.123:0/2617778979 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b54014d90 con 0x7f7b6c0f5410 2024-01-13T11:52:27.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.906+0000 7f7b711ab700 1 -- 172.21.15.123:0/2617778979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b6c1061b0 con 0x7f7b6c0f5410 2024-01-13T11:52:27.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.907+0000 7f7b637fe700 1 -- 172.21.15.123:0/2617778979 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b540095f0 con 0x7f7b6c0f5410 2024-01-13T11:52:27.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.907+0000 7f7b711ab700 1 -- 172.21.15.123:0/2617778979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6c045ad0 con 0x7f7b6c0f5410 2024-01-13T11:52:27.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.907+0000 7f7b637fe700 1 --2- 172.21.15.123:0/2617778979 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7b5802ff60 0x7f7b58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:27.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.907+0000 7f7b637fe700 1 -- 172.21.15.123:0/2617778979 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b54041080 con 0x7f7b6c0f5410 2024-01-13T11:52:27.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.907+0000 7f7b6a59c700 1 --2- 172.21.15.123:0/2617778979 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7b5802ff60 0x7f7b58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:27.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.908+0000 7f7b6a59c700 1 --2- 172.21.15.123:0/2617778979 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7b5802ff60 0x7f7b58032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f7b5c0096a0 tx=0x7f7b5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:27.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:27.912+0000 7f7b637fe700 1 -- 172.21.15.123:0/2617778979 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b54014420 con 0x7f7b6c0f5410 2024-01-13T11:52:28.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:28.119+0000 7f7b711ab700 1 -- 172.21.15.123:0/2617778979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b6c0fae60 con 0x7f7b6c0f5410 2024-01-13T11:52:28.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:28.120+0000 7f7b637fe700 1 -- 172.21.15.123:0/2617778979 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7b54010230 con 0x7f7b6c0f5410 2024-01-13T11:52:28.109 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:28.109 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:28.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:28.121+0000 7f7b711ab700 1 -- 172.21.15.123:0/2617778979 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7b5802ff60 msgr2=0x7f7b58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:28.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:28.121+0000 7f7b711ab700 1 --2- 172.21.15.123:0/2617778979 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7b5802ff60 0x7f7b58032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f7b5c0096a0 tx=0x7f7b5c006b40).stop 2024-01-13T11:52:28.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:28.121+0000 7f7b711ab700 1 -- 172.21.15.123:0/2617778979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b6c0f5410 msgr2=0x7f7b6c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:28.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:28.121+0000 7f7b711ab700 1 --2- 172.21.15.123:0/2617778979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b6c0f5410 0x7f7b6c105630 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f7b54013510 tx=0x7f7b5400b040).stop 2024-01-13T11:52:28.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:28.122+0000 7f7b711ab700 1 -- 172.21.15.123:0/2617778979 shutdown_connections 2024-01-13T11:52:28.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:28.122+0000 7f7b711ab700 1 --2- 172.21.15.123:0/2617778979 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7b5802ff60 0x7f7b58032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:28.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:28.122+0000 7f7b711ab700 1 --2- 172.21.15.123:0/2617778979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b6c0f5410 0x7f7b6c105630 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:28.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:28.122+0000 7f7b711ab700 1 -- 172.21.15.123:0/2617778979 >> 172.21.15.123:0/2617778979 conn(0x7f7b6c0f0b70 msgr2=0x7f7b6c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:28.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:28.122+0000 7f7b711ab700 1 -- 172.21.15.123:0/2617778979 shutdown_connections 2024-01-13T11:52:28.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:28.122+0000 7f7b711ab700 1 -- 172.21.15.123:0/2617778979 wait complete. 2024-01-13T11:52:28.112 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:28.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:28 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:28.121243+0000 mon.smithi059 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.123:0/2617778979' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:29.533 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:29.533 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:29.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:29 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:27.525710+0000 mgr.smithi059.hmyldj (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:30.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.712+0000 7f7cfc893700 1 -- 172.21.15.123:0/3073278537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf80f51a0 msgr2=0x7f7cf80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:30.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.712+0000 7f7cfc893700 1 --2- 172.21.15.123:0/3073278537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf80f51a0 0x7f7cf80f55c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f7ce8009fb0 tx=0x7f7ce8009b30).stop 2024-01-13T11:52:30.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.712+0000 7f7cfc893700 1 -- 172.21.15.123:0/3073278537 shutdown_connections 2024-01-13T11:52:30.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.712+0000 7f7cfc893700 1 --2- 172.21.15.123:0/3073278537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf80f51a0 0x7f7cf80f55c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:30.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.712+0000 7f7cfc893700 1 -- 172.21.15.123:0/3073278537 >> 172.21.15.123:0/3073278537 conn(0x7f7cf80f09a0 msgr2=0x7f7cf80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:30.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.712+0000 7f7cfc893700 1 -- 172.21.15.123:0/3073278537 shutdown_connections 2024-01-13T11:52:30.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.712+0000 7f7cfc893700 1 -- 172.21.15.123:0/3073278537 wait complete. 2024-01-13T11:52:30.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.713+0000 7f7cfc893700 1 Processor -- start 2024-01-13T11:52:30.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.713+0000 7f7cfc893700 1 -- start start 2024-01-13T11:52:30.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.713+0000 7f7cfc893700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf80f51a0 0x7f7cf81091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:30.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.713+0000 7f7cfc893700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf8060220 con 0x7f7cf80f51a0 2024-01-13T11:52:30.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.713+0000 7f7cf77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf80f51a0 0x7f7cf81091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:30.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.713+0000 7f7cf77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf80f51a0 0x7f7cf81091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:34828/0 (socket says 172.21.15.123:34828) 2024-01-13T11:52:30.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.713+0000 7f7cf77fe700 1 -- 172.21.15.123:0/1207736738 learned_addr learned my addr 172.21.15.123:0/1207736738 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:30.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.714+0000 7f7cf77fe700 1 -- 172.21.15.123:0/1207736738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce8005040 con 0x7f7cf80f51a0 2024-01-13T11:52:30.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.714+0000 7f7cf77fe700 1 --2- 172.21.15.123:0/1207736738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf80f51a0 0x7f7cf81091d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f7ce8013510 tx=0x7f7ce800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:30.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.714+0000 7f7cdffff700 1 -- 172.21.15.123:0/1207736738 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ce8009320 con 0x7f7cf80f51a0 2024-01-13T11:52:30.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.714+0000 7f7cdffff700 1 -- 172.21.15.123:0/1207736738 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ce8013f00 con 0x7f7cf80f51a0 2024-01-13T11:52:30.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.714+0000 7f7cfc893700 1 -- 172.21.15.123:0/1207736738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf8060420 con 0x7f7cf80f51a0 2024-01-13T11:52:30.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.714+0000 7f7cdffff700 1 -- 172.21.15.123:0/1207736738 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ce8014dc0 con 0x7f7cf80f51a0 2024-01-13T11:52:30.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.714+0000 7f7cfc893700 1 -- 172.21.15.123:0/1207736738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf8109880 con 0x7f7cf80f51a0 2024-01-13T11:52:30.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.715+0000 7f7cdffff700 1 -- 172.21.15.123:0/1207736738 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ce80095f0 con 0x7f7cf80f51a0 2024-01-13T11:52:30.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.715+0000 7f7cdffff700 1 --2- 172.21.15.123:0/1207736738 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7ce002ff60 0x7f7ce0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:30.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.715+0000 7f7cfc893700 1 -- 172.21.15.123:0/1207736738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf8059100 con 0x7f7cf80f51a0 2024-01-13T11:52:30.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.715+0000 7f7cdffff700 1 -- 172.21.15.123:0/1207736738 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ce8041100 con 0x7f7cf80f51a0 2024-01-13T11:52:30.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.715+0000 7f7cf6ffd700 1 --2- 172.21.15.123:0/1207736738 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7ce002ff60 0x7f7ce0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:30.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.715+0000 7f7cf6ffd700 1 --2- 172.21.15.123:0/1207736738 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7ce002ff60 0x7f7ce0032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f7cec0096a0 tx=0x7f7cec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:30.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.720+0000 7f7cdffff700 1 -- 172.21.15.123:0/1207736738 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ce801c790 con 0x7f7cf80f51a0 2024-01-13T11:52:30.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.924+0000 7f7cfc893700 1 -- 172.21.15.123:0/1207736738 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7cf80fabe0 con 0x7f7cf80f51a0 2024-01-13T11:52:30.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.924+0000 7f7cdffff700 1 -- 172.21.15.123:0/1207736738 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7ce8010230 con 0x7f7cf80f51a0 2024-01-13T11:52:30.913 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:30.914 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:30.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.926+0000 7f7cfc893700 1 -- 172.21.15.123:0/1207736738 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7ce002ff60 msgr2=0x7f7ce0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:30.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.926+0000 7f7cfc893700 1 --2- 172.21.15.123:0/1207736738 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7ce002ff60 0x7f7ce0032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f7cec0096a0 tx=0x7f7cec006b40).stop 2024-01-13T11:52:30.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.926+0000 7f7cfc893700 1 -- 172.21.15.123:0/1207736738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf80f51a0 msgr2=0x7f7cf81091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:30.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.926+0000 7f7cfc893700 1 --2- 172.21.15.123:0/1207736738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf80f51a0 0x7f7cf81091d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f7ce8013510 tx=0x7f7ce800b040).stop 2024-01-13T11:52:30.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.926+0000 7f7cfc893700 1 -- 172.21.15.123:0/1207736738 shutdown_connections 2024-01-13T11:52:30.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.926+0000 7f7cfc893700 1 --2- 172.21.15.123:0/1207736738 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7ce002ff60 0x7f7ce0032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:30.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.926+0000 7f7cfc893700 1 --2- 172.21.15.123:0/1207736738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cf80f51a0 0x7f7cf81091d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:30.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.926+0000 7f7cfc893700 1 -- 172.21.15.123:0/1207736738 >> 172.21.15.123:0/1207736738 conn(0x7f7cf80f09a0 msgr2=0x7f7cf80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:30.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.926+0000 7f7cfc893700 1 -- 172.21.15.123:0/1207736738 shutdown_connections 2024-01-13T11:52:30.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:30.926+0000 7f7cfc893700 1 -- 172.21.15.123:0/1207736738 wait complete. 2024-01-13T11:52:30.916 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:31.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:31 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:29.525938+0000 mgr.smithi059.hmyldj (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:31.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:31 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:30.925595+0000 mon.smithi059 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.123:0/1207736738' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:32.306 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:32.307 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:33.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.437+0000 7f2636f5a700 1 -- 172.21.15.123:0/3394481231 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26300f6510 msgr2=0x7f26300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:33.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.437+0000 7f2636f5a700 1 --2- 172.21.15.123:0/3394481231 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26300f6510 0x7f26300f6930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f2624009fb0 tx=0x7f2624009b30).stop 2024-01-13T11:52:33.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.437+0000 7f2636f5a700 1 -- 172.21.15.123:0/3394481231 shutdown_connections 2024-01-13T11:52:33.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.437+0000 7f2636f5a700 1 --2- 172.21.15.123:0/3394481231 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26300f6510 0x7f26300f6930 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:33.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.437+0000 7f2636f5a700 1 -- 172.21.15.123:0/3394481231 >> 172.21.15.123:0/3394481231 conn(0x7f26300f1c70 msgr2=0x7f26300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:33.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.437+0000 7f2636f5a700 1 -- 172.21.15.123:0/3394481231 shutdown_connections 2024-01-13T11:52:33.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.437+0000 7f2636f5a700 1 -- 172.21.15.123:0/3394481231 wait complete. 2024-01-13T11:52:33.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.437+0000 7f2636f5a700 1 Processor -- start 2024-01-13T11:52:33.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.438+0000 7f2636f5a700 1 -- start start 2024-01-13T11:52:33.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.438+0000 7f2636f5a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26300f6510 0x7f2630106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:33.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.438+0000 7f2636f5a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2630106c20 con 0x7f26300f6510 2024-01-13T11:52:33.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.438+0000 7f2634cf6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26300f6510 0x7f2630106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:33.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.438+0000 7f2634cf6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26300f6510 0x7f2630106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:34838/0 (socket says 172.21.15.123:34838) 2024-01-13T11:52:33.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.438+0000 7f2634cf6700 1 -- 172.21.15.123:0/2555302470 learned_addr learned my addr 172.21.15.123:0/2555302470 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:33.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.438+0000 7f2634cf6700 1 -- 172.21.15.123:0/2555302470 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2624005040 con 0x7f26300f6510 2024-01-13T11:52:33.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.438+0000 7f2634cf6700 1 --2- 172.21.15.123:0/2555302470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26300f6510 0x7f2630106730 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f2624013510 tx=0x7f262400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:33.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.439+0000 7f262d7fa700 1 -- 172.21.15.123:0/2555302470 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2624009320 con 0x7f26300f6510 2024-01-13T11:52:33.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.439+0000 7f2636f5a700 1 -- 172.21.15.123:0/2555302470 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2630106e20 con 0x7f26300f6510 2024-01-13T11:52:33.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.439+0000 7f262d7fa700 1 -- 172.21.15.123:0/2555302470 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2624013f00 con 0x7f26300f6510 2024-01-13T11:52:33.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.439+0000 7f262d7fa700 1 -- 172.21.15.123:0/2555302470 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2624014d90 con 0x7f26300f6510 2024-01-13T11:52:33.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.439+0000 7f2636f5a700 1 -- 172.21.15.123:0/2555302470 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26301072b0 con 0x7f26300f6510 2024-01-13T11:52:33.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.439+0000 7f262d7fa700 1 -- 172.21.15.123:0/2555302470 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2624014420 con 0x7f26300f6510 2024-01-13T11:52:33.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.439+0000 7f2636f5a700 1 -- 172.21.15.123:0/2555302470 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2630045ad0 con 0x7f26300f6510 2024-01-13T11:52:33.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.440+0000 7f262d7fa700 1 --2- 172.21.15.123:0/2555302470 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f262002ff60 0x7f2620032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:33.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.440+0000 7f262d7fa700 1 -- 172.21.15.123:0/2555302470 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f26240410b0 con 0x7f26300f6510 2024-01-13T11:52:33.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.440+0000 7f262ffff700 1 --2- 172.21.15.123:0/2555302470 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f262002ff60 0x7f2620032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:33.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.440+0000 7f262ffff700 1 --2- 172.21.15.123:0/2555302470 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f262002ff60 0x7f2620032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f261c0096a0 tx=0x7f261c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:33.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.445+0000 7f262d7fa700 1 -- 172.21.15.123:0/2555302470 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26240095f0 con 0x7f26300f6510 2024-01-13T11:52:33.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:33 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:31.526258+0000 mgr.smithi059.hmyldj (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:33.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.653+0000 7f2636f5a700 1 -- 172.21.15.123:0/2555302470 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26300fbf60 con 0x7f26300f6510 2024-01-13T11:52:33.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.654+0000 7f262d7fa700 1 -- 172.21.15.123:0/2555302470 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f262401db70 con 0x7f26300f6510 2024-01-13T11:52:33.642 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:33.642 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:33.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.655+0000 7f2636f5a700 1 -- 172.21.15.123:0/2555302470 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f262002ff60 msgr2=0x7f2620032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:33.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.655+0000 7f2636f5a700 1 --2- 172.21.15.123:0/2555302470 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f262002ff60 0x7f2620032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f261c0096a0 tx=0x7f261c006b40).stop 2024-01-13T11:52:33.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.655+0000 7f2636f5a700 1 -- 172.21.15.123:0/2555302470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26300f6510 msgr2=0x7f2630106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:33.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.655+0000 7f2636f5a700 1 --2- 172.21.15.123:0/2555302470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26300f6510 0x7f2630106730 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f2624013510 tx=0x7f262400b040).stop 2024-01-13T11:52:33.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.655+0000 7f2636f5a700 1 -- 172.21.15.123:0/2555302470 shutdown_connections 2024-01-13T11:52:33.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.655+0000 7f2636f5a700 1 --2- 172.21.15.123:0/2555302470 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f262002ff60 0x7f2620032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:33.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.655+0000 7f2636f5a700 1 --2- 172.21.15.123:0/2555302470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26300f6510 0x7f2630106730 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:33.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.655+0000 7f2636f5a700 1 -- 172.21.15.123:0/2555302470 >> 172.21.15.123:0/2555302470 conn(0x7f26300f1c70 msgr2=0x7f26300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:33.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.655+0000 7f2636f5a700 1 -- 172.21.15.123:0/2555302470 shutdown_connections 2024-01-13T11:52:33.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:33.655+0000 7f2636f5a700 1 -- 172.21.15.123:0/2555302470 wait complete. 2024-01-13T11:52:33.645 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:34.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:34 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:33.654060+0000 mon.smithi059 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.123:0/2555302470' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:35.048 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:35.048 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:35.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:35 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:33.526662+0000 mgr.smithi059.hmyldj (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:36.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.214+0000 7f21314ac700 1 -- 172.21.15.123:0/954238848 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f212c0f6510 msgr2=0x7f212c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:36.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.214+0000 7f21314ac700 1 --2- 172.21.15.123:0/954238848 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f212c0f6510 0x7f212c0f6930 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f2120009fb0 tx=0x7f2120009b30).stop 2024-01-13T11:52:36.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.214+0000 7f21314ac700 1 -- 172.21.15.123:0/954238848 shutdown_connections 2024-01-13T11:52:36.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.214+0000 7f21314ac700 1 --2- 172.21.15.123:0/954238848 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f212c0f6510 0x7f212c0f6930 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:36.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.214+0000 7f21314ac700 1 -- 172.21.15.123:0/954238848 >> 172.21.15.123:0/954238848 conn(0x7f212c0f1c70 msgr2=0x7f212c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:36.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.214+0000 7f21314ac700 1 -- 172.21.15.123:0/954238848 shutdown_connections 2024-01-13T11:52:36.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.214+0000 7f21314ac700 1 -- 172.21.15.123:0/954238848 wait complete. 2024-01-13T11:52:36.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.215+0000 7f21314ac700 1 Processor -- start 2024-01-13T11:52:36.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.215+0000 7f21314ac700 1 -- start start 2024-01-13T11:52:36.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.215+0000 7f21314ac700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f212c0f6510 0x7f212c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:36.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.215+0000 7f21314ac700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212c106c20 con 0x7f212c0f6510 2024-01-13T11:52:36.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.215+0000 7f212affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f212c0f6510 0x7f212c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:36.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.215+0000 7f212affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f212c0f6510 0x7f212c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:34842/0 (socket says 172.21.15.123:34842) 2024-01-13T11:52:36.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.215+0000 7f212affd700 1 -- 172.21.15.123:0/2612402423 learned_addr learned my addr 172.21.15.123:0/2612402423 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:36.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.216+0000 7f212affd700 1 -- 172.21.15.123:0/2612402423 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2120005040 con 0x7f212c0f6510 2024-01-13T11:52:36.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.216+0000 7f212affd700 1 --2- 172.21.15.123:0/2612402423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f212c0f6510 0x7f212c106730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f2120013510 tx=0x7f212000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:36.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.216+0000 7f21137fe700 1 -- 172.21.15.123:0/2612402423 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2120009320 con 0x7f212c0f6510 2024-01-13T11:52:36.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.216+0000 7f21314ac700 1 -- 172.21.15.123:0/2612402423 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f212c106e20 con 0x7f212c0f6510 2024-01-13T11:52:36.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.216+0000 7f21137fe700 1 -- 172.21.15.123:0/2612402423 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2120013f00 con 0x7f212c0f6510 2024-01-13T11:52:36.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.216+0000 7f21314ac700 1 -- 172.21.15.123:0/2612402423 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f212c1072b0 con 0x7f212c0f6510 2024-01-13T11:52:36.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.216+0000 7f21137fe700 1 -- 172.21.15.123:0/2612402423 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2120014d90 con 0x7f212c0f6510 2024-01-13T11:52:36.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.217+0000 7f21137fe700 1 -- 172.21.15.123:0/2612402423 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f21200095f0 con 0x7f212c0f6510 2024-01-13T11:52:36.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.217+0000 7f21314ac700 1 -- 172.21.15.123:0/2612402423 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f212c045ad0 con 0x7f212c0f6510 2024-01-13T11:52:36.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.217+0000 7f21137fe700 1 --2- 172.21.15.123:0/2612402423 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f211402ff60 0x7f2114032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:36.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.217+0000 7f21137fe700 1 -- 172.21.15.123:0/2612402423 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2120041090 con 0x7f212c0f6510 2024-01-13T11:52:36.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.217+0000 7f212a7fc700 1 --2- 172.21.15.123:0/2612402423 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f211402ff60 0x7f2114032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:36.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.218+0000 7f212a7fc700 1 --2- 172.21.15.123:0/2612402423 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f211402ff60 0x7f2114032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f211c0096a0 tx=0x7f211c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:36.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.222+0000 7f21137fe700 1 -- 172.21.15.123:0/2612402423 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2120014420 con 0x7f212c0f6510 2024-01-13T11:52:36.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.429+0000 7f21314ac700 1 -- 172.21.15.123:0/2612402423 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f212c0fbf60 con 0x7f212c0f6510 2024-01-13T11:52:36.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.429+0000 7f21137fe700 1 -- 172.21.15.123:0/2612402423 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2120010230 con 0x7f212c0f6510 2024-01-13T11:52:36.419 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:36.419 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:36.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.431+0000 7f21314ac700 1 -- 172.21.15.123:0/2612402423 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f211402ff60 msgr2=0x7f2114032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:36.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.431+0000 7f21314ac700 1 --2- 172.21.15.123:0/2612402423 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f211402ff60 0x7f2114032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f211c0096a0 tx=0x7f211c006b40).stop 2024-01-13T11:52:36.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.431+0000 7f21314ac700 1 -- 172.21.15.123:0/2612402423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f212c0f6510 msgr2=0x7f212c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:36.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.431+0000 7f21314ac700 1 --2- 172.21.15.123:0/2612402423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f212c0f6510 0x7f212c106730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f2120013510 tx=0x7f212000b040).stop 2024-01-13T11:52:36.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.431+0000 7f21314ac700 1 -- 172.21.15.123:0/2612402423 shutdown_connections 2024-01-13T11:52:36.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.431+0000 7f21314ac700 1 --2- 172.21.15.123:0/2612402423 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f211402ff60 0x7f2114032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:36.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.431+0000 7f21314ac700 1 --2- 172.21.15.123:0/2612402423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f212c0f6510 0x7f212c106730 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:36.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.431+0000 7f21314ac700 1 -- 172.21.15.123:0/2612402423 >> 172.21.15.123:0/2612402423 conn(0x7f212c0f1c70 msgr2=0x7f212c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:36.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.431+0000 7f21314ac700 1 -- 172.21.15.123:0/2612402423 shutdown_connections 2024-01-13T11:52:36.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:36.431+0000 7f21314ac700 1 -- 172.21.15.123:0/2612402423 wait complete. 2024-01-13T11:52:36.421 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:37.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:37 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:35.527067+0000 mgr.smithi059.hmyldj (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:37.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:37 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:36.430587+0000 mon.smithi059 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.123:0/2612402423' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:37.806 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:37.806 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:38.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.000+0000 7f21c3818700 1 -- 172.21.15.123:0/3326444107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21bc0f47c0 msgr2=0x7f21bc0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:38.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.000+0000 7f21c3818700 1 --2- 172.21.15.123:0/3326444107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21bc0f47c0 0x7f21bc0f4be0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f21ac009fb0 tx=0x7f21ac009b30).stop 2024-01-13T11:52:38.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.000+0000 7f21c3818700 1 -- 172.21.15.123:0/3326444107 shutdown_connections 2024-01-13T11:52:38.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.000+0000 7f21c3818700 1 --2- 172.21.15.123:0/3326444107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21bc0f47c0 0x7f21bc0f4be0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:38.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.000+0000 7f21c3818700 1 -- 172.21.15.123:0/3326444107 >> 172.21.15.123:0/3326444107 conn(0x7f21bc0f02e0 msgr2=0x7f21bc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:38.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.001+0000 7f21c3818700 1 -- 172.21.15.123:0/3326444107 shutdown_connections 2024-01-13T11:52:38.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.001+0000 7f21c3818700 1 -- 172.21.15.123:0/3326444107 wait complete. 2024-01-13T11:52:38.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.001+0000 7f21c3818700 1 Processor -- start 2024-01-13T11:52:38.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.001+0000 7f21c3818700 1 -- start start 2024-01-13T11:52:38.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.001+0000 7f21c3818700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21bc0f47c0 0x7f21bc106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:38.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.001+0000 7f21c3818700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21bc106bf0 con 0x7f21bc0f47c0 2024-01-13T11:52:38.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.001+0000 7f21c15b4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21bc0f47c0 0x7f21bc106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:38.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.001+0000 7f21c15b4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21bc0f47c0 0x7f21bc106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:37978/0 (socket says 172.21.15.123:37978) 2024-01-13T11:52:38.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.001+0000 7f21c15b4700 1 -- 172.21.15.123:0/1931499958 learned_addr learned my addr 172.21.15.123:0/1931499958 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:38.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.002+0000 7f21c15b4700 1 -- 172.21.15.123:0/1931499958 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ac005040 con 0x7f21bc0f47c0 2024-01-13T11:52:38.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.002+0000 7f21c15b4700 1 --2- 172.21.15.123:0/1931499958 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21bc0f47c0 0x7f21bc106700 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f21ac013510 tx=0x7f21ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:38.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.002+0000 7f21b1ffb700 1 -- 172.21.15.123:0/1931499958 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21ac009320 con 0x7f21bc0f47c0 2024-01-13T11:52:38.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.002+0000 7f21c3818700 1 -- 172.21.15.123:0/1931499958 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21bc106df0 con 0x7f21bc0f47c0 2024-01-13T11:52:38.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.002+0000 7f21b1ffb700 1 -- 172.21.15.123:0/1931499958 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21ac013f00 con 0x7f21bc0f47c0 2024-01-13T11:52:38.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.002+0000 7f21b1ffb700 1 -- 172.21.15.123:0/1931499958 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21ac014d90 con 0x7f21bc0f47c0 2024-01-13T11:52:38.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.002+0000 7f21c3818700 1 -- 172.21.15.123:0/1931499958 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21bc107280 con 0x7f21bc0f47c0 2024-01-13T11:52:38.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.003+0000 7f21b1ffb700 1 -- 172.21.15.123:0/1931499958 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f21ac0263a0 con 0x7f21bc0f47c0 2024-01-13T11:52:38.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.003+0000 7f21b1ffb700 1 --2- 172.21.15.123:0/1931499958 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f21a802ff60 0x7f21a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:38.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.003+0000 7f21b1ffb700 1 -- 172.21.15.123:0/1931499958 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f21ac0410a0 con 0x7f21bc0f47c0 2024-01-13T11:52:38.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.003+0000 7f21c3818700 1 -- 172.21.15.123:0/1931499958 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21bc059100 con 0x7f21bc0f47c0 2024-01-13T11:52:38.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.003+0000 7f21c0db3700 1 --2- 172.21.15.123:0/1931499958 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f21a802ff60 0x7f21a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:38.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.004+0000 7f21c0db3700 1 --2- 172.21.15.123:0/1931499958 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f21a802ff60 0x7f21a8032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f21b80096a0 tx=0x7f21b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:38.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.008+0000 7f21b1ffb700 1 -- 172.21.15.123:0/1931499958 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f21ac012920 con 0x7f21bc0f47c0 2024-01-13T11:52:39.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.215+0000 7f21c3818700 1 -- 172.21.15.123:0/1931499958 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21bc045ad0 con 0x7f21bc0f47c0 2024-01-13T11:52:39.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.215+0000 7f21b1ffb700 1 -- 172.21.15.123:0/1931499958 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f21ac0126f0 con 0x7f21bc0f47c0 2024-01-13T11:52:39.204 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:39.205 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:39.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.217+0000 7f21c3818700 1 -- 172.21.15.123:0/1931499958 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f21a802ff60 msgr2=0x7f21a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:39.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.217+0000 7f21c3818700 1 --2- 172.21.15.123:0/1931499958 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f21a802ff60 0x7f21a8032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f21b80096a0 tx=0x7f21b8006b40).stop 2024-01-13T11:52:39.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.217+0000 7f21c3818700 1 -- 172.21.15.123:0/1931499958 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21bc0f47c0 msgr2=0x7f21bc106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:39.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.217+0000 7f21c3818700 1 --2- 172.21.15.123:0/1931499958 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21bc0f47c0 0x7f21bc106700 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f21ac013510 tx=0x7f21ac00b040).stop 2024-01-13T11:52:39.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.217+0000 7f21c3818700 1 -- 172.21.15.123:0/1931499958 shutdown_connections 2024-01-13T11:52:39.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.217+0000 7f21c3818700 1 --2- 172.21.15.123:0/1931499958 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f21a802ff60 0x7f21a8032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:39.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.217+0000 7f21c3818700 1 --2- 172.21.15.123:0/1931499958 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21bc0f47c0 0x7f21bc106700 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:39.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.217+0000 7f21c3818700 1 -- 172.21.15.123:0/1931499958 >> 172.21.15.123:0/1931499958 conn(0x7f21bc0f02e0 msgr2=0x7f21bc0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:39.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.217+0000 7f21c3818700 1 -- 172.21.15.123:0/1931499958 shutdown_connections 2024-01-13T11:52:39.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:39.217+0000 7f21c3818700 1 -- 172.21.15.123:0/1931499958 wait complete. 2024-01-13T11:52:39.207 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:39.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:39 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:37.527352+0000 mgr.smithi059.hmyldj (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:40.591 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:40.592 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:40.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:39.216558+0000 mon.smithi059 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.123:0/1931499958' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:40.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:39.639239+0000 mon.smithi059 (mon.0) 173 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/mirror_snapshot_schedule"}]: dispatch 2024-01-13T11:52:40.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:39.641303+0000 mon.smithi059 (mon.0) 174 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/trash_purge_schedule"}]: dispatch 2024-01-13T11:52:41.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:41 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:39.527686+0000 mgr.smithi059.hmyldj (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:41.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.799+0000 7f1e77906700 1 -- 172.21.15.123:0/3682540811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e700f5410 msgr2=0x7f1e700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:41.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.799+0000 7f1e77906700 1 --2- 172.21.15.123:0/3682540811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e700f5410 0x7f1e700f5830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f1e6c009fb0 tx=0x7f1e6c009b30).stop 2024-01-13T11:52:41.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.799+0000 7f1e77906700 1 -- 172.21.15.123:0/3682540811 shutdown_connections 2024-01-13T11:52:41.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.799+0000 7f1e77906700 1 --2- 172.21.15.123:0/3682540811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e700f5410 0x7f1e700f5830 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:41.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.799+0000 7f1e77906700 1 -- 172.21.15.123:0/3682540811 >> 172.21.15.123:0/3682540811 conn(0x7f1e700f0b70 msgr2=0x7f1e700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:41.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.799+0000 7f1e77906700 1 -- 172.21.15.123:0/3682540811 shutdown_connections 2024-01-13T11:52:41.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.799+0000 7f1e77906700 1 -- 172.21.15.123:0/3682540811 wait complete. 2024-01-13T11:52:41.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.799+0000 7f1e77906700 1 Processor -- start 2024-01-13T11:52:41.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.800+0000 7f1e77906700 1 -- start start 2024-01-13T11:52:41.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.800+0000 7f1e77906700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e70060200 0x7f1e70109560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:41.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.800+0000 7f1e77906700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e70109a50 con 0x7f1e70060200 2024-01-13T11:52:41.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.800+0000 7f1e756a2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e70060200 0x7f1e70109560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:41.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.800+0000 7f1e756a2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e70060200 0x7f1e70109560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:37990/0 (socket says 172.21.15.123:37990) 2024-01-13T11:52:41.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.800+0000 7f1e756a2700 1 -- 172.21.15.123:0/3704834484 learned_addr learned my addr 172.21.15.123:0/3704834484 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:41.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.800+0000 7f1e756a2700 1 -- 172.21.15.123:0/3704834484 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c005040 con 0x7f1e70060200 2024-01-13T11:52:41.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.800+0000 7f1e756a2700 1 --2- 172.21.15.123:0/3704834484 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e70060200 0x7f1e70109560 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f1e6c009f80 tx=0x7f1e6c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:41.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.800+0000 7f1e65ffb700 1 -- 172.21.15.123:0/3704834484 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e6c0136c0 con 0x7f1e70060200 2024-01-13T11:52:41.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.801+0000 7f1e77906700 1 -- 172.21.15.123:0/3704834484 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e70109b90 con 0x7f1e70060200 2024-01-13T11:52:41.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.801+0000 7f1e65ffb700 1 -- 172.21.15.123:0/3704834484 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e6c013e70 con 0x7f1e70060200 2024-01-13T11:52:41.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.801+0000 7f1e65ffb700 1 -- 172.21.15.123:0/3704834484 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e6c014420 con 0x7f1e70060200 2024-01-13T11:52:41.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.801+0000 7f1e77906700 1 -- 172.21.15.123:0/3704834484 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e70109fc0 con 0x7f1e70060200 2024-01-13T11:52:41.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.801+0000 7f1e65ffb700 1 -- 172.21.15.123:0/3704834484 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e6c025060 con 0x7f1e70060200 2024-01-13T11:52:41.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.802+0000 7f1e65ffb700 1 --2- 172.21.15.123:0/3704834484 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1e5c02ff60 0x7f1e5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:41.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.802+0000 7f1e77906700 1 -- 172.21.15.123:0/3704834484 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e70059100 con 0x7f1e70060200 2024-01-13T11:52:41.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.802+0000 7f1e65ffb700 1 -- 172.21.15.123:0/3704834484 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e6c040ef0 con 0x7f1e70060200 2024-01-13T11:52:41.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.802+0000 7f1e74ea1700 1 --2- 172.21.15.123:0/3704834484 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1e5c02ff60 0x7f1e5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:41.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.802+0000 7f1e74ea1700 1 --2- 172.21.15.123:0/3704834484 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1e5c02ff60 0x7f1e5c032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f1e600096a0 tx=0x7f1e60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:41.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:41.807+0000 7f1e65ffb700 1 -- 172.21.15.123:0/3704834484 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e6c028440 con 0x7f1e70060200 2024-01-13T11:52:41.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:42.010+0000 7f1e77906700 1 -- 172.21.15.123:0/3704834484 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e700fae60 con 0x7f1e70060200 2024-01-13T11:52:42.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:42.011+0000 7f1e65ffb700 1 -- 172.21.15.123:0/3704834484 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1e6c004490 con 0x7f1e70060200 2024-01-13T11:52:42.000 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:42.000 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:42.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:42.012+0000 7f1e77906700 1 -- 172.21.15.123:0/3704834484 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1e5c02ff60 msgr2=0x7f1e5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:42.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:42.012+0000 7f1e77906700 1 --2- 172.21.15.123:0/3704834484 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1e5c02ff60 0x7f1e5c032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f1e600096a0 tx=0x7f1e60006b40).stop 2024-01-13T11:52:42.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:42.012+0000 7f1e77906700 1 -- 172.21.15.123:0/3704834484 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e70060200 msgr2=0x7f1e70109560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:42.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:42.012+0000 7f1e77906700 1 --2- 172.21.15.123:0/3704834484 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e70060200 0x7f1e70109560 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f1e6c009f80 tx=0x7f1e6c009b30).stop 2024-01-13T11:52:42.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:42.012+0000 7f1e77906700 1 -- 172.21.15.123:0/3704834484 shutdown_connections 2024-01-13T11:52:42.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:42.012+0000 7f1e77906700 1 --2- 172.21.15.123:0/3704834484 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1e5c02ff60 0x7f1e5c032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:42.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:42.012+0000 7f1e77906700 1 --2- 172.21.15.123:0/3704834484 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e70060200 0x7f1e70109560 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:42.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:42.012+0000 7f1e77906700 1 -- 172.21.15.123:0/3704834484 >> 172.21.15.123:0/3704834484 conn(0x7f1e700f0b70 msgr2=0x7f1e700f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:42.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:42.012+0000 7f1e77906700 1 -- 172.21.15.123:0/3704834484 shutdown_connections 2024-01-13T11:52:42.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:42.012+0000 7f1e77906700 1 -- 172.21.15.123:0/3704834484 wait complete. 2024-01-13T11:52:42.002 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:42.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:42 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:42.012027+0000 mon.smithi059 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.123:0/3704834484' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:43.390 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:43.391 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:43.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:43 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:41.528026+0000 mgr.smithi059.hmyldj (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:44.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.565+0000 7fd8391f0700 1 -- 172.21.15.123:0/799373359 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8340f6510 msgr2=0x7fd8340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:44.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.565+0000 7fd8391f0700 1 --2- 172.21.15.123:0/799373359 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8340f6510 0x7fd8340f6930 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fd824009fb0 tx=0x7fd824009b30).stop 2024-01-13T11:52:44.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.565+0000 7fd8391f0700 1 -- 172.21.15.123:0/799373359 shutdown_connections 2024-01-13T11:52:44.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.565+0000 7fd8391f0700 1 --2- 172.21.15.123:0/799373359 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8340f6510 0x7fd8340f6930 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:44.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.565+0000 7fd8391f0700 1 -- 172.21.15.123:0/799373359 >> 172.21.15.123:0/799373359 conn(0x7fd8340f1c70 msgr2=0x7fd8340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:44.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.565+0000 7fd8391f0700 1 -- 172.21.15.123:0/799373359 shutdown_connections 2024-01-13T11:52:44.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.565+0000 7fd8391f0700 1 -- 172.21.15.123:0/799373359 wait complete. 2024-01-13T11:52:44.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.565+0000 7fd8391f0700 1 Processor -- start 2024-01-13T11:52:44.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.565+0000 7fd8391f0700 1 -- start start 2024-01-13T11:52:44.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.566+0000 7fd8391f0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8340f6510 0x7fd834106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:44.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.566+0000 7fd8391f0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd834106c20 con 0x7fd8340f6510 2024-01-13T11:52:44.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.566+0000 7fd832d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8340f6510 0x7fd834106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:44.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.566+0000 7fd832d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8340f6510 0x7fd834106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:38006/0 (socket says 172.21.15.123:38006) 2024-01-13T11:52:44.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.566+0000 7fd832d9d700 1 -- 172.21.15.123:0/4182630355 learned_addr learned my addr 172.21.15.123:0/4182630355 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:44.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.566+0000 7fd832d9d700 1 -- 172.21.15.123:0/4182630355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd824005040 con 0x7fd8340f6510 2024-01-13T11:52:44.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.566+0000 7fd832d9d700 1 --2- 172.21.15.123:0/4182630355 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8340f6510 0x7fd834106730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fd824013510 tx=0x7fd82400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:44.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.566+0000 7fd81b7fe700 1 -- 172.21.15.123:0/4182630355 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd824009320 con 0x7fd8340f6510 2024-01-13T11:52:44.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.566+0000 7fd8391f0700 1 -- 172.21.15.123:0/4182630355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd834106e20 con 0x7fd8340f6510 2024-01-13T11:52:44.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.566+0000 7fd81b7fe700 1 -- 172.21.15.123:0/4182630355 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd824013f00 con 0x7fd8340f6510 2024-01-13T11:52:44.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.566+0000 7fd81b7fe700 1 -- 172.21.15.123:0/4182630355 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd824014d90 con 0x7fd8340f6510 2024-01-13T11:52:44.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.566+0000 7fd8391f0700 1 -- 172.21.15.123:0/4182630355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8341072b0 con 0x7fd8340f6510 2024-01-13T11:52:44.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.567+0000 7fd81b7fe700 1 -- 172.21.15.123:0/4182630355 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd8240263a0 con 0x7fd8340f6510 2024-01-13T11:52:44.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.567+0000 7fd81b7fe700 1 --2- 172.21.15.123:0/4182630355 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd81c02ff60 0x7fd81c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:44.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.567+0000 7fd81b7fe700 1 -- 172.21.15.123:0/4182630355 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd824041010 con 0x7fd8340f6510 2024-01-13T11:52:44.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.567+0000 7fd8391f0700 1 -- 172.21.15.123:0/4182630355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd834045ad0 con 0x7fd8340f6510 2024-01-13T11:52:44.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.568+0000 7fd83259c700 1 --2- 172.21.15.123:0/4182630355 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd81c02ff60 0x7fd81c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:44.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.568+0000 7fd83259c700 1 --2- 172.21.15.123:0/4182630355 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd81c02ff60 0x7fd81c032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fd8280096a0 tx=0x7fd828006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:44.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.573+0000 7fd81b7fe700 1 -- 172.21.15.123:0/4182630355 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd824012920 con 0x7fd8340f6510 2024-01-13T11:52:44.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.782+0000 7fd8391f0700 1 -- 172.21.15.123:0/4182630355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd8340fbf60 con 0x7fd8340f6510 2024-01-13T11:52:44.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.782+0000 7fd81b7fe700 1 -- 172.21.15.123:0/4182630355 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd82402b020 con 0x7fd8340f6510 2024-01-13T11:52:44.772 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:44.773 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:44.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.784+0000 7fd8391f0700 1 -- 172.21.15.123:0/4182630355 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd81c02ff60 msgr2=0x7fd81c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:44.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.784+0000 7fd8391f0700 1 --2- 172.21.15.123:0/4182630355 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd81c02ff60 0x7fd81c032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fd8280096a0 tx=0x7fd828006b40).stop 2024-01-13T11:52:44.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.784+0000 7fd8391f0700 1 -- 172.21.15.123:0/4182630355 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8340f6510 msgr2=0x7fd834106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:44.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.784+0000 7fd8391f0700 1 --2- 172.21.15.123:0/4182630355 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8340f6510 0x7fd834106730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fd824013510 tx=0x7fd82400b040).stop 2024-01-13T11:52:44.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.784+0000 7fd8391f0700 1 -- 172.21.15.123:0/4182630355 shutdown_connections 2024-01-13T11:52:44.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.784+0000 7fd8391f0700 1 --2- 172.21.15.123:0/4182630355 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd81c02ff60 0x7fd81c032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:44.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.784+0000 7fd8391f0700 1 --2- 172.21.15.123:0/4182630355 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8340f6510 0x7fd834106730 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:44.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.784+0000 7fd8391f0700 1 -- 172.21.15.123:0/4182630355 >> 172.21.15.123:0/4182630355 conn(0x7fd8340f1c70 msgr2=0x7fd8340f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:44.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.784+0000 7fd8391f0700 1 -- 172.21.15.123:0/4182630355 shutdown_connections 2024-01-13T11:52:44.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:44.784+0000 7fd8391f0700 1 -- 172.21.15.123:0/4182630355 wait complete. 2024-01-13T11:52:44.775 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:45.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:45 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:43.528362+0000 mgr.smithi059.hmyldj (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:45.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:45 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:44.783620+0000 mon.smithi059 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.123:0/4182630355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:46.166 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:46.166 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:47.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.344+0000 7fa4eac8e700 1 -- 172.21.15.123:0/229113680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e40f6510 msgr2=0x7fa4e40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:47.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.344+0000 7fa4eac8e700 1 --2- 172.21.15.123:0/229113680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e40f6510 0x7fa4e40f6930 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fa4d8009fb0 tx=0x7fa4d8009b30).stop 2024-01-13T11:52:47.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.344+0000 7fa4eac8e700 1 -- 172.21.15.123:0/229113680 shutdown_connections 2024-01-13T11:52:47.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.344+0000 7fa4eac8e700 1 --2- 172.21.15.123:0/229113680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e40f6510 0x7fa4e40f6930 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:47.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.344+0000 7fa4eac8e700 1 -- 172.21.15.123:0/229113680 >> 172.21.15.123:0/229113680 conn(0x7fa4e40f1c70 msgr2=0x7fa4e40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:47.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.345+0000 7fa4eac8e700 1 -- 172.21.15.123:0/229113680 shutdown_connections 2024-01-13T11:52:47.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.345+0000 7fa4eac8e700 1 -- 172.21.15.123:0/229113680 wait complete. 2024-01-13T11:52:47.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.345+0000 7fa4eac8e700 1 Processor -- start 2024-01-13T11:52:47.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.345+0000 7fa4eac8e700 1 -- start start 2024-01-13T11:52:47.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.345+0000 7fa4eac8e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e40f6510 0x7fa4e406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:47.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.345+0000 7fa4eac8e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e4069e20 con 0x7fa4e40f6510 2024-01-13T11:52:47.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.346+0000 7fa4e8a2a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e40f6510 0x7fa4e406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:47.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.346+0000 7fa4e8a2a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e40f6510 0x7fa4e406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:37242/0 (socket says 172.21.15.123:37242) 2024-01-13T11:52:47.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.346+0000 7fa4e8a2a700 1 -- 172.21.15.123:0/992261591 learned_addr learned my addr 172.21.15.123:0/992261591 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:47.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.346+0000 7fa4e8a2a700 1 -- 172.21.15.123:0/992261591 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d8005040 con 0x7fa4e40f6510 2024-01-13T11:52:47.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.346+0000 7fa4e8a2a700 1 --2- 172.21.15.123:0/992261591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e40f6510 0x7fa4e406b9a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fa4d8013510 tx=0x7fa4d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:47.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.346+0000 7fa4e17fa700 1 -- 172.21.15.123:0/992261591 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4d8009320 con 0x7fa4e40f6510 2024-01-13T11:52:47.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.346+0000 7fa4eac8e700 1 -- 172.21.15.123:0/992261591 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e4069fc0 con 0x7fa4e40f6510 2024-01-13T11:52:47.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.346+0000 7fa4e17fa700 1 -- 172.21.15.123:0/992261591 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa4d8013f00 con 0x7fa4e40f6510 2024-01-13T11:52:47.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.346+0000 7fa4e17fa700 1 -- 172.21.15.123:0/992261591 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4d8014d90 con 0x7fa4e40f6510 2024-01-13T11:52:47.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.346+0000 7fa4eac8e700 1 -- 172.21.15.123:0/992261591 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e406a450 con 0x7fa4e40f6510 2024-01-13T11:52:47.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.347+0000 7fa4e17fa700 1 -- 172.21.15.123:0/992261591 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa4d80263a0 con 0x7fa4e40f6510 2024-01-13T11:52:47.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.347+0000 7fa4e17fa700 1 --2- 172.21.15.123:0/992261591 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa4d402ff60 0x7fa4d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:47.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.347+0000 7fa4e17fa700 1 -- 172.21.15.123:0/992261591 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa4d8040ff0 con 0x7fa4e40f6510 2024-01-13T11:52:47.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.347+0000 7fa4eac8e700 1 -- 172.21.15.123:0/992261591 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e4059100 con 0x7fa4e40f6510 2024-01-13T11:52:47.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.347+0000 7fa4e3fff700 1 --2- 172.21.15.123:0/992261591 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa4d402ff60 0x7fa4d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:47.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.348+0000 7fa4e3fff700 1 --2- 172.21.15.123:0/992261591 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa4d402ff60 0x7fa4d4032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fa4d00096a0 tx=0x7fa4d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:47.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.352+0000 7fa4e17fa700 1 -- 172.21.15.123:0/992261591 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa4d8012920 con 0x7fa4e40f6510 2024-01-13T11:52:47.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.558+0000 7fa4eac8e700 1 -- 172.21.15.123:0/992261591 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4e406ab50 con 0x7fa4e40f6510 2024-01-13T11:52:47.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.558+0000 7fa4e17fa700 1 -- 172.21.15.123:0/992261591 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa4d802b020 con 0x7fa4e40f6510 2024-01-13T11:52:47.548 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:47.548 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:47.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.560+0000 7fa4eac8e700 1 -- 172.21.15.123:0/992261591 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa4d402ff60 msgr2=0x7fa4d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:47.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.560+0000 7fa4eac8e700 1 --2- 172.21.15.123:0/992261591 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa4d402ff60 0x7fa4d4032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fa4d00096a0 tx=0x7fa4d0006b40).stop 2024-01-13T11:52:47.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.560+0000 7fa4eac8e700 1 -- 172.21.15.123:0/992261591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e40f6510 msgr2=0x7fa4e406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:47.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.560+0000 7fa4eac8e700 1 --2- 172.21.15.123:0/992261591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e40f6510 0x7fa4e406b9a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fa4d8013510 tx=0x7fa4d800b040).stop 2024-01-13T11:52:47.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.560+0000 7fa4eac8e700 1 -- 172.21.15.123:0/992261591 shutdown_connections 2024-01-13T11:52:47.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.560+0000 7fa4eac8e700 1 --2- 172.21.15.123:0/992261591 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa4d402ff60 0x7fa4d4032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:47.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.560+0000 7fa4eac8e700 1 --2- 172.21.15.123:0/992261591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4e40f6510 0x7fa4e406b9a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:47.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.560+0000 7fa4eac8e700 1 -- 172.21.15.123:0/992261591 >> 172.21.15.123:0/992261591 conn(0x7fa4e40f1c70 msgr2=0x7fa4e40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:47.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.560+0000 7fa4eac8e700 1 -- 172.21.15.123:0/992261591 shutdown_connections 2024-01-13T11:52:47.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:47.560+0000 7fa4eac8e700 1 -- 172.21.15.123:0/992261591 wait complete. 2024-01-13T11:52:47.550 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:47.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:47 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:45.528602+0000 mgr.smithi059.hmyldj (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:48.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:48 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:47.559803+0000 mon.smithi059 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.123:0/992261591' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:48.944 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:48.945 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:49.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:49 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:47.528959+0000 mgr.smithi059.hmyldj (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:50.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.122+0000 7f7288483700 1 -- 172.21.15.123:0/308912841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7280060280 msgr2=0x7f72800f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:50.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.122+0000 7f7288483700 1 --2- 172.21.15.123:0/308912841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7280060280 0x7f72800f90f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f7270009fb0 tx=0x7f7270009b30).stop 2024-01-13T11:52:50.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.122+0000 7f7288483700 1 -- 172.21.15.123:0/308912841 shutdown_connections 2024-01-13T11:52:50.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.122+0000 7f7288483700 1 --2- 172.21.15.123:0/308912841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7280060280 0x7f72800f90f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:50.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.122+0000 7f7288483700 1 -- 172.21.15.123:0/308912841 >> 172.21.15.123:0/308912841 conn(0x7f72800f0b70 msgr2=0x7f72800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:50.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.122+0000 7f7288483700 1 -- 172.21.15.123:0/308912841 shutdown_connections 2024-01-13T11:52:50.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.122+0000 7f7288483700 1 -- 172.21.15.123:0/308912841 wait complete. 2024-01-13T11:52:50.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.123+0000 7f7288483700 1 Processor -- start 2024-01-13T11:52:50.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.123+0000 7f7288483700 1 -- start start 2024-01-13T11:52:50.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.123+0000 7f7288483700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7280060280 0x7f72801066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:50.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.123+0000 7f7288483700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7280106bd0 con 0x7f7280060280 2024-01-13T11:52:50.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.123+0000 7f728621f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7280060280 0x7f72801066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:50.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.123+0000 7f728621f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7280060280 0x7f72801066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:37270/0 (socket says 172.21.15.123:37270) 2024-01-13T11:52:50.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.123+0000 7f728621f700 1 -- 172.21.15.123:0/3096614217 learned_addr learned my addr 172.21.15.123:0/3096614217 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:50.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.123+0000 7f728621f700 1 -- 172.21.15.123:0/3096614217 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7270005040 con 0x7f7280060280 2024-01-13T11:52:50.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.124+0000 7f728621f700 1 --2- 172.21.15.123:0/3096614217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7280060280 0x7f72801066e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f7270013510 tx=0x7f727000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:50.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.124+0000 7f7276ffd700 1 -- 172.21.15.123:0/3096614217 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7270009320 con 0x7f7280060280 2024-01-13T11:52:50.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.124+0000 7f7288483700 1 -- 172.21.15.123:0/3096614217 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7280106dd0 con 0x7f7280060280 2024-01-13T11:52:50.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.124+0000 7f7276ffd700 1 -- 172.21.15.123:0/3096614217 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7270013f00 con 0x7f7280060280 2024-01-13T11:52:50.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.124+0000 7f7276ffd700 1 -- 172.21.15.123:0/3096614217 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7270014dc0 con 0x7f7280060280 2024-01-13T11:52:50.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.124+0000 7f7288483700 1 -- 172.21.15.123:0/3096614217 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7280107260 con 0x7f7280060280 2024-01-13T11:52:50.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.125+0000 7f7276ffd700 1 -- 172.21.15.123:0/3096614217 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72700095f0 con 0x7f7280060280 2024-01-13T11:52:50.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.125+0000 7f7288483700 1 -- 172.21.15.123:0/3096614217 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7280045ad0 con 0x7f7280060280 2024-01-13T11:52:50.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.125+0000 7f7276ffd700 1 --2- 172.21.15.123:0/3096614217 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f726c02ff60 0x7f726c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:50.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.125+0000 7f7276ffd700 1 -- 172.21.15.123:0/3096614217 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7270041000 con 0x7f7280060280 2024-01-13T11:52:50.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.125+0000 7f7285a1e700 1 --2- 172.21.15.123:0/3096614217 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f726c02ff60 0x7f726c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:50.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.125+0000 7f7285a1e700 1 --2- 172.21.15.123:0/3096614217 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f726c02ff60 0x7f726c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f727c0096a0 tx=0x7f727c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:50.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.129+0000 7f7276ffd700 1 -- 172.21.15.123:0/3096614217 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7270014420 con 0x7f7280060280 2024-01-13T11:52:50.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.337+0000 7f7288483700 1 -- 172.21.15.123:0/3096614217 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7280108110 con 0x7f7280060280 2024-01-13T11:52:50.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.338+0000 7f7276ffd700 1 -- 172.21.15.123:0/3096614217 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7270010230 con 0x7f7280060280 2024-01-13T11:52:50.327 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:50.327 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:50.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.340+0000 7f7288483700 1 -- 172.21.15.123:0/3096614217 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f726c02ff60 msgr2=0x7f726c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:50.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.340+0000 7f7288483700 1 --2- 172.21.15.123:0/3096614217 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f726c02ff60 0x7f726c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f727c0096a0 tx=0x7f727c006b40).stop 2024-01-13T11:52:50.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.340+0000 7f7288483700 1 -- 172.21.15.123:0/3096614217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7280060280 msgr2=0x7f72801066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:50.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.340+0000 7f7288483700 1 --2- 172.21.15.123:0/3096614217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7280060280 0x7f72801066e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f7270013510 tx=0x7f727000b040).stop 2024-01-13T11:52:50.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.340+0000 7f7288483700 1 -- 172.21.15.123:0/3096614217 shutdown_connections 2024-01-13T11:52:50.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.340+0000 7f7288483700 1 --2- 172.21.15.123:0/3096614217 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f726c02ff60 0x7f726c032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:50.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.340+0000 7f7288483700 1 --2- 172.21.15.123:0/3096614217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7280060280 0x7f72801066e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:50.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.340+0000 7f7288483700 1 -- 172.21.15.123:0/3096614217 >> 172.21.15.123:0/3096614217 conn(0x7f72800f0b70 msgr2=0x7f72800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:50.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.340+0000 7f7288483700 1 -- 172.21.15.123:0/3096614217 shutdown_connections 2024-01-13T11:52:50.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:50.340+0000 7f7288483700 1 -- 172.21.15.123:0/3096614217 wait complete. 2024-01-13T11:52:50.330 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:51.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:49.529294+0000 mgr.smithi059.hmyldj (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:51.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:50.339364+0000 mon.smithi059 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.123:0/3096614217' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:51.738 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:51.738 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:52.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.891+0000 7f719825a700 1 -- 172.21.15.123:0/3701560492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71900f5410 msgr2=0x7f71900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:52.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.891+0000 7f719825a700 1 --2- 172.21.15.123:0/3701560492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71900f5410 0x7f71900f5830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f718c009fb0 tx=0x7f718c009b30).stop 2024-01-13T11:52:52.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.891+0000 7f719825a700 1 -- 172.21.15.123:0/3701560492 shutdown_connections 2024-01-13T11:52:52.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.891+0000 7f719825a700 1 --2- 172.21.15.123:0/3701560492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71900f5410 0x7f71900f5830 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:52.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.891+0000 7f719825a700 1 -- 172.21.15.123:0/3701560492 >> 172.21.15.123:0/3701560492 conn(0x7f71900f0b70 msgr2=0x7f71900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:52.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.891+0000 7f719825a700 1 -- 172.21.15.123:0/3701560492 shutdown_connections 2024-01-13T11:52:52.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.891+0000 7f719825a700 1 -- 172.21.15.123:0/3701560492 wait complete. 2024-01-13T11:52:52.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.891+0000 7f719825a700 1 Processor -- start 2024-01-13T11:52:52.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.891+0000 7f719825a700 1 -- start start 2024-01-13T11:52:52.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.892+0000 7f719825a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71900f5410 0x7f71900fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:52.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.892+0000 7f719825a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71900fb820 con 0x7f71900f5410 2024-01-13T11:52:52.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.892+0000 7f7195ff6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71900f5410 0x7f71900fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:52.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.892+0000 7f7195ff6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71900f5410 0x7f71900fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:37290/0 (socket says 172.21.15.123:37290) 2024-01-13T11:52:52.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.892+0000 7f7195ff6700 1 -- 172.21.15.123:0/1392500948 learned_addr learned my addr 172.21.15.123:0/1392500948 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:52.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.892+0000 7f7195ff6700 1 -- 172.21.15.123:0/1392500948 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f718c005040 con 0x7f71900f5410 2024-01-13T11:52:52.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.892+0000 7f7195ff6700 1 --2- 172.21.15.123:0/1392500948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71900f5410 0x7f71900fd3a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f718c013510 tx=0x7f718c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:52.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.892+0000 7f71867fc700 1 -- 172.21.15.123:0/1392500948 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f718c009320 con 0x7f71900f5410 2024-01-13T11:52:52.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.892+0000 7f719825a700 1 -- 172.21.15.123:0/1392500948 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71900fb9c0 con 0x7f71900f5410 2024-01-13T11:52:52.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.892+0000 7f71867fc700 1 -- 172.21.15.123:0/1392500948 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f718c013f00 con 0x7f71900f5410 2024-01-13T11:52:52.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.892+0000 7f71867fc700 1 -- 172.21.15.123:0/1392500948 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f718c014d90 con 0x7f71900f5410 2024-01-13T11:52:52.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.893+0000 7f719825a700 1 -- 172.21.15.123:0/1392500948 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71900fbe50 con 0x7f71900f5410 2024-01-13T11:52:52.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.893+0000 7f71867fc700 1 -- 172.21.15.123:0/1392500948 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f718c0095f0 con 0x7f71900f5410 2024-01-13T11:52:52.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.893+0000 7f719825a700 1 -- 172.21.15.123:0/1392500948 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7190059100 con 0x7f71900f5410 2024-01-13T11:52:52.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.893+0000 7f71867fc700 1 --2- 172.21.15.123:0/1392500948 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f717c02ff60 0x7f717c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:52.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.893+0000 7f71867fc700 1 -- 172.21.15.123:0/1392500948 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f718c041000 con 0x7f71900f5410 2024-01-13T11:52:52.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.894+0000 7f71957f5700 1 --2- 172.21.15.123:0/1392500948 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f717c02ff60 0x7f717c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:52.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.894+0000 7f71957f5700 1 --2- 172.21.15.123:0/1392500948 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f717c02ff60 0x7f717c032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f71800096a0 tx=0x7f7180006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:52.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:52.898+0000 7f71867fc700 1 -- 172.21.15.123:0/1392500948 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f718c014420 con 0x7f71900f5410 2024-01-13T11:52:53.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:53.106+0000 7f719825a700 1 -- 172.21.15.123:0/1392500948 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71900fae60 con 0x7f71900f5410 2024-01-13T11:52:53.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:53.107+0000 7f71867fc700 1 -- 172.21.15.123:0/1392500948 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f718c010230 con 0x7f71900f5410 2024-01-13T11:52:53.096 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:53.096 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:53.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:53.109+0000 7f719825a700 1 -- 172.21.15.123:0/1392500948 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f717c02ff60 msgr2=0x7f717c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:53.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:53.109+0000 7f719825a700 1 --2- 172.21.15.123:0/1392500948 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f717c02ff60 0x7f717c032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f71800096a0 tx=0x7f7180006b40).stop 2024-01-13T11:52:53.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:53.109+0000 7f719825a700 1 -- 172.21.15.123:0/1392500948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71900f5410 msgr2=0x7f71900fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:53.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:53.109+0000 7f719825a700 1 --2- 172.21.15.123:0/1392500948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71900f5410 0x7f71900fd3a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f718c013510 tx=0x7f718c00b040).stop 2024-01-13T11:52:53.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:53.109+0000 7f719825a700 1 -- 172.21.15.123:0/1392500948 shutdown_connections 2024-01-13T11:52:53.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:53.109+0000 7f719825a700 1 --2- 172.21.15.123:0/1392500948 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f717c02ff60 0x7f717c032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:53.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:53.109+0000 7f719825a700 1 --2- 172.21.15.123:0/1392500948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71900f5410 0x7f71900fd3a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:53.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:53.109+0000 7f719825a700 1 -- 172.21.15.123:0/1392500948 >> 172.21.15.123:0/1392500948 conn(0x7f71900f0b70 msgr2=0x7f71900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:53.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:53.109+0000 7f719825a700 1 -- 172.21.15.123:0/1392500948 shutdown_connections 2024-01-13T11:52:53.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:53.109+0000 7f719825a700 1 -- 172.21.15.123:0/1392500948 wait complete. 2024-01-13T11:52:53.099 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:53.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:53 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:51.529679+0000 mgr.smithi059.hmyldj (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:53.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:53 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:53.108168+0000 mon.smithi059 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.123:0/1392500948' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:54.509 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:54.510 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:55.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:55 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:53.530019+0000 mgr.smithi059.hmyldj (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:55.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.667+0000 7fb217576700 1 -- 172.21.15.123:0/2044813557 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2100601e0 msgr2=0x7fb2100f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:55.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.667+0000 7fb217576700 1 --2- 172.21.15.123:0/2044813557 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2100601e0 0x7fb2100f8f20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fb20c009fb0 tx=0x7fb20c009b30).stop 2024-01-13T11:52:55.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.667+0000 7fb217576700 1 -- 172.21.15.123:0/2044813557 shutdown_connections 2024-01-13T11:52:55.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.667+0000 7fb217576700 1 --2- 172.21.15.123:0/2044813557 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2100601e0 0x7fb2100f8f20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:55.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.667+0000 7fb217576700 1 -- 172.21.15.123:0/2044813557 >> 172.21.15.123:0/2044813557 conn(0x7fb2100f09a0 msgr2=0x7fb2100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:55.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.667+0000 7fb217576700 1 -- 172.21.15.123:0/2044813557 shutdown_connections 2024-01-13T11:52:55.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.667+0000 7fb217576700 1 -- 172.21.15.123:0/2044813557 wait complete. 2024-01-13T11:52:55.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.667+0000 7fb217576700 1 Processor -- start 2024-01-13T11:52:55.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.667+0000 7fb217576700 1 -- start start 2024-01-13T11:52:55.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.667+0000 7fb217576700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2100601e0 0x7fb210106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:55.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.667+0000 7fb217576700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2101069f0 con 0x7fb2100601e0 2024-01-13T11:52:55.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.668+0000 7fb216574700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2100601e0 0x7fb210106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:55.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.668+0000 7fb216574700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2100601e0 0x7fb210106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:37298/0 (socket says 172.21.15.123:37298) 2024-01-13T11:52:55.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.668+0000 7fb216574700 1 -- 172.21.15.123:0/1265378529 learned_addr learned my addr 172.21.15.123:0/1265378529 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:55.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.668+0000 7fb216574700 1 -- 172.21.15.123:0/1265378529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20c005040 con 0x7fb2100601e0 2024-01-13T11:52:55.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.668+0000 7fb216574700 1 --2- 172.21.15.123:0/1265378529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2100601e0 0x7fb210106500 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fb20c013510 tx=0x7fb20c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:55.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.668+0000 7fb206ffd700 1 -- 172.21.15.123:0/1265378529 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb20c009320 con 0x7fb2100601e0 2024-01-13T11:52:55.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.668+0000 7fb206ffd700 1 -- 172.21.15.123:0/1265378529 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb20c013f00 con 0x7fb2100601e0 2024-01-13T11:52:55.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.668+0000 7fb217576700 1 -- 172.21.15.123:0/1265378529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb210106bf0 con 0x7fb2100601e0 2024-01-13T11:52:55.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.668+0000 7fb206ffd700 1 -- 172.21.15.123:0/1265378529 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb20c014d90 con 0x7fb2100601e0 2024-01-13T11:52:55.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.668+0000 7fb217576700 1 -- 172.21.15.123:0/1265378529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb210107080 con 0x7fb2100601e0 2024-01-13T11:52:55.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.669+0000 7fb206ffd700 1 -- 172.21.15.123:0/1265378529 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb20c0263a0 con 0x7fb2100601e0 2024-01-13T11:52:55.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.669+0000 7fb206ffd700 1 --2- 172.21.15.123:0/1265378529 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb1fc02ff60 0x7fb1fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:55.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.669+0000 7fb217576700 1 -- 172.21.15.123:0/1265378529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb210045ad0 con 0x7fb2100601e0 2024-01-13T11:52:55.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.669+0000 7fb206ffd700 1 -- 172.21.15.123:0/1265378529 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb20c041330 con 0x7fb2100601e0 2024-01-13T11:52:55.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.670+0000 7fb215d73700 1 --2- 172.21.15.123:0/1265378529 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb1fc02ff60 0x7fb1fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:55.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.670+0000 7fb215d73700 1 --2- 172.21.15.123:0/1265378529 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb1fc02ff60 0x7fb1fc032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fb2000096a0 tx=0x7fb200006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:55.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.674+0000 7fb206ffd700 1 -- 172.21.15.123:0/1265378529 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb20c012920 con 0x7fb2100601e0 2024-01-13T11:52:55.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.882+0000 7fb217576700 1 -- 172.21.15.123:0/1265378529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb210107f30 con 0x7fb2100601e0 2024-01-13T11:52:55.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.882+0000 7fb206ffd700 1 -- 172.21.15.123:0/1265378529 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb20c012330 con 0x7fb2100601e0 2024-01-13T11:52:55.871 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:55.871 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:55.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.884+0000 7fb217576700 1 -- 172.21.15.123:0/1265378529 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb1fc02ff60 msgr2=0x7fb1fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:55.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.884+0000 7fb217576700 1 --2- 172.21.15.123:0/1265378529 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb1fc02ff60 0x7fb1fc032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fb2000096a0 tx=0x7fb200006b40).stop 2024-01-13T11:52:55.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.884+0000 7fb217576700 1 -- 172.21.15.123:0/1265378529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2100601e0 msgr2=0x7fb210106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:55.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.884+0000 7fb217576700 1 --2- 172.21.15.123:0/1265378529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2100601e0 0x7fb210106500 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fb20c013510 tx=0x7fb20c00b040).stop 2024-01-13T11:52:55.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.884+0000 7fb217576700 1 -- 172.21.15.123:0/1265378529 shutdown_connections 2024-01-13T11:52:55.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.884+0000 7fb217576700 1 --2- 172.21.15.123:0/1265378529 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb1fc02ff60 0x7fb1fc032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:55.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.884+0000 7fb217576700 1 --2- 172.21.15.123:0/1265378529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2100601e0 0x7fb210106500 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:55.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.884+0000 7fb217576700 1 -- 172.21.15.123:0/1265378529 >> 172.21.15.123:0/1265378529 conn(0x7fb2100f09a0 msgr2=0x7fb2100f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:55.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.884+0000 7fb217576700 1 -- 172.21.15.123:0/1265378529 shutdown_connections 2024-01-13T11:52:55.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:55.884+0000 7fb217576700 1 -- 172.21.15.123:0/1265378529 wait complete. 2024-01-13T11:52:55.874 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:56.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:56 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:55.883667+0000 mon.smithi059 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.123:0/1265378529' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:52:57.265 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:52:57.266 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:52:57.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:55.530288+0000 mgr.smithi059.hmyldj (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:58.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.422+0000 7fe50ce82700 1 -- 172.21.15.123:0/3872719872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe50806b960 msgr2=0x7fe508069e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:58.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.422+0000 7fe50ce82700 1 --2- 172.21.15.123:0/3872719872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe50806b960 0x7fe508069e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fe4fc009fb0 tx=0x7fe4fc009b30).stop 2024-01-13T11:52:58.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.422+0000 7fe50ce82700 1 -- 172.21.15.123:0/3872719872 shutdown_connections 2024-01-13T11:52:58.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.422+0000 7fe50ce82700 1 --2- 172.21.15.123:0/3872719872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe50806b960 0x7fe508069e60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:58.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.422+0000 7fe50ce82700 1 -- 172.21.15.123:0/3872719872 >> 172.21.15.123:0/3872719872 conn(0x7fe5080f1c70 msgr2=0x7fe5080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:58.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.422+0000 7fe50ce82700 1 -- 172.21.15.123:0/3872719872 shutdown_connections 2024-01-13T11:52:58.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.422+0000 7fe50ce82700 1 -- 172.21.15.123:0/3872719872 wait complete. 2024-01-13T11:52:58.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.423+0000 7fe50ce82700 1 Processor -- start 2024-01-13T11:52:58.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.423+0000 7fe50ce82700 1 -- start start 2024-01-13T11:52:58.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.423+0000 7fe50ce82700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe50806b960 0x7fe5081088a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:58.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.423+0000 7fe50ce82700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe508108d90 con 0x7fe50806b960 2024-01-13T11:52:58.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.423+0000 7fe50659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe50806b960 0x7fe5081088a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:58.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.423+0000 7fe50659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe50806b960 0x7fe5081088a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:36658/0 (socket says 172.21.15.123:36658) 2024-01-13T11:52:58.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.423+0000 7fe50659c700 1 -- 172.21.15.123:0/3358627252 learned_addr learned my addr 172.21.15.123:0/3358627252 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:52:58.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.424+0000 7fe50659c700 1 -- 172.21.15.123:0/3358627252 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc005040 con 0x7fe50806b960 2024-01-13T11:52:58.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.424+0000 7fe50659c700 1 --2- 172.21.15.123:0/3358627252 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe50806b960 0x7fe5081088a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fe4fc013510 tx=0x7fe4fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:58.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.424+0000 7fe4f6ffd700 1 -- 172.21.15.123:0/3358627252 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4fc009320 con 0x7fe50806b960 2024-01-13T11:52:58.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.424+0000 7fe50ce82700 1 -- 172.21.15.123:0/3358627252 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe508108f90 con 0x7fe50806b960 2024-01-13T11:52:58.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.424+0000 7fe4f6ffd700 1 -- 172.21.15.123:0/3358627252 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4fc013f00 con 0x7fe50806b960 2024-01-13T11:52:58.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.424+0000 7fe4f6ffd700 1 -- 172.21.15.123:0/3358627252 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4fc014dc0 con 0x7fe50806b960 2024-01-13T11:52:58.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.424+0000 7fe50ce82700 1 -- 172.21.15.123:0/3358627252 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe508109420 con 0x7fe50806b960 2024-01-13T11:52:58.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.425+0000 7fe4f6ffd700 1 -- 172.21.15.123:0/3358627252 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe4fc014420 con 0x7fe50806b960 2024-01-13T11:52:58.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.425+0000 7fe50ce82700 1 -- 172.21.15.123:0/3358627252 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe508045ad0 con 0x7fe50806b960 2024-01-13T11:52:58.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.425+0000 7fe4f6ffd700 1 --2- 172.21.15.123:0/3358627252 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe4f002ff60 0x7fe4f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:52:58.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.425+0000 7fe4f6ffd700 1 -- 172.21.15.123:0/3358627252 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe4fc041050 con 0x7fe50806b960 2024-01-13T11:52:58.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.425+0000 7fe505d9b700 1 --2- 172.21.15.123:0/3358627252 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe4f002ff60 0x7fe4f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:52:58.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.426+0000 7fe505d9b700 1 --2- 172.21.15.123:0/3358627252 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe4f002ff60 0x7fe4f0032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fe4f80096a0 tx=0x7fe4f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:52:58.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.430+0000 7fe4f6ffd700 1 -- 172.21.15.123:0/3358627252 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe4fc0095f0 con 0x7fe50806b960 2024-01-13T11:52:58.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.638+0000 7fe50ce82700 1 -- 172.21.15.123:0/3358627252 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe50810a2d0 con 0x7fe50806b960 2024-01-13T11:52:58.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.639+0000 7fe4f6ffd700 1 -- 172.21.15.123:0/3358627252 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe4fc01cb10 con 0x7fe50806b960 2024-01-13T11:52:58.629 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:52:58.629 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:52:58.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.641+0000 7fe50ce82700 1 -- 172.21.15.123:0/3358627252 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe4f002ff60 msgr2=0x7fe4f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:58.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.641+0000 7fe50ce82700 1 --2- 172.21.15.123:0/3358627252 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe4f002ff60 0x7fe4f0032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fe4f80096a0 tx=0x7fe4f8006b40).stop 2024-01-13T11:52:58.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.641+0000 7fe50ce82700 1 -- 172.21.15.123:0/3358627252 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe50806b960 msgr2=0x7fe5081088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:52:58.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.641+0000 7fe50ce82700 1 --2- 172.21.15.123:0/3358627252 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe50806b960 0x7fe5081088a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fe4fc013510 tx=0x7fe4fc00b040).stop 2024-01-13T11:52:58.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.641+0000 7fe50ce82700 1 -- 172.21.15.123:0/3358627252 shutdown_connections 2024-01-13T11:52:58.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.641+0000 7fe50ce82700 1 --2- 172.21.15.123:0/3358627252 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe4f002ff60 0x7fe4f0032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:58.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.641+0000 7fe50ce82700 1 --2- 172.21.15.123:0/3358627252 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe50806b960 0x7fe5081088a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:52:58.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.641+0000 7fe50ce82700 1 -- 172.21.15.123:0/3358627252 >> 172.21.15.123:0/3358627252 conn(0x7fe5080f1c70 msgr2=0x7fe5080f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:52:58.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.641+0000 7fe50ce82700 1 -- 172.21.15.123:0/3358627252 shutdown_connections 2024-01-13T11:52:58.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:52:58.641+0000 7fe50ce82700 1 -- 172.21.15.123:0/3358627252 wait complete. 2024-01-13T11:52:58.631 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:52:59.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:59 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:57.530597+0000 mgr.smithi059.hmyldj (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:59.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:52:59 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:52:58.640522+0000 mon.smithi059 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.123:0/3358627252' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:00.031 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:00.031 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:01.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.171+0000 7f8be6d75700 1 -- 172.21.15.123:0/2677247957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8be00f51a0 msgr2=0x7f8be00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:01.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.171+0000 7f8be6d75700 1 --2- 172.21.15.123:0/2677247957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8be00f51a0 0x7f8be00f55c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f8bdc009fb0 tx=0x7f8bdc009b30).stop 2024-01-13T11:53:01.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.171+0000 7f8be6d75700 1 -- 172.21.15.123:0/2677247957 shutdown_connections 2024-01-13T11:53:01.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.171+0000 7f8be6d75700 1 --2- 172.21.15.123:0/2677247957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8be00f51a0 0x7f8be00f55c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:01.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.171+0000 7f8be6d75700 1 -- 172.21.15.123:0/2677247957 >> 172.21.15.123:0/2677247957 conn(0x7f8be00f09a0 msgr2=0x7f8be00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:01.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.171+0000 7f8be6d75700 1 -- 172.21.15.123:0/2677247957 shutdown_connections 2024-01-13T11:53:01.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.171+0000 7f8be6d75700 1 -- 172.21.15.123:0/2677247957 wait complete. 2024-01-13T11:53:01.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.171+0000 7f8be6d75700 1 Processor -- start 2024-01-13T11:53:01.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.172+0000 7f8be6d75700 1 -- start start 2024-01-13T11:53:01.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.172+0000 7f8be6d75700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8be00f51a0 0x7f8be01064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:01.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.172+0000 7f8be6d75700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be0106990 con 0x7f8be00f51a0 2024-01-13T11:53:01.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.172+0000 7f8be5d73700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8be00f51a0 0x7f8be01064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:01.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.172+0000 7f8be5d73700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8be00f51a0 0x7f8be01064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:36684/0 (socket says 172.21.15.123:36684) 2024-01-13T11:53:01.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.172+0000 7f8be5d73700 1 -- 172.21.15.123:0/2553129032 learned_addr learned my addr 172.21.15.123:0/2553129032 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:01.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.172+0000 7f8be5d73700 1 -- 172.21.15.123:0/2553129032 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bdc005040 con 0x7f8be00f51a0 2024-01-13T11:53:01.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.172+0000 7f8be5d73700 1 --2- 172.21.15.123:0/2553129032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8be00f51a0 0x7f8be01064a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f8bdc013510 tx=0x7f8bdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:01.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.173+0000 7f8bd67fc700 1 -- 172.21.15.123:0/2553129032 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bdc009320 con 0x7f8be00f51a0 2024-01-13T11:53:01.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.173+0000 7f8be6d75700 1 -- 172.21.15.123:0/2553129032 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8be0106b90 con 0x7f8be00f51a0 2024-01-13T11:53:01.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.173+0000 7f8bd67fc700 1 -- 172.21.15.123:0/2553129032 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8bdc013f00 con 0x7f8be00f51a0 2024-01-13T11:53:01.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.173+0000 7f8bd67fc700 1 -- 172.21.15.123:0/2553129032 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bdc014dc0 con 0x7f8be00f51a0 2024-01-13T11:53:01.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.173+0000 7f8be6d75700 1 -- 172.21.15.123:0/2553129032 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8be0107020 con 0x7f8be00f51a0 2024-01-13T11:53:01.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.173+0000 7f8bd67fc700 1 -- 172.21.15.123:0/2553129032 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8bdc0095f0 con 0x7f8be00f51a0 2024-01-13T11:53:01.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.173+0000 7f8bd67fc700 1 --2- 172.21.15.123:0/2553129032 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8bcc02ff60 0x7f8bcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:01.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.173+0000 7f8be6d75700 1 -- 172.21.15.123:0/2553129032 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8be0045ad0 con 0x7f8be00f51a0 2024-01-13T11:53:01.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.174+0000 7f8bd67fc700 1 -- 172.21.15.123:0/2553129032 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8bdc0410e0 con 0x7f8be00f51a0 2024-01-13T11:53:01.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.174+0000 7f8be5572700 1 --2- 172.21.15.123:0/2553129032 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8bcc02ff60 0x7f8bcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:01.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.174+0000 7f8be5572700 1 --2- 172.21.15.123:0/2553129032 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8bcc02ff60 0x7f8bcc032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f8bd00096a0 tx=0x7f8bd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:01.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.178+0000 7f8bd67fc700 1 -- 172.21.15.123:0/2553129032 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8bdc06d2b0 con 0x7f8be00f51a0 2024-01-13T11:53:01.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.388+0000 7f8be6d75700 1 -- 172.21.15.123:0/2553129032 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8be0107e70 con 0x7f8be00f51a0 2024-01-13T11:53:01.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.389+0000 7f8bd67fc700 1 -- 172.21.15.123:0/2553129032 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8bdc01c920 con 0x7f8be00f51a0 2024-01-13T11:53:01.378 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:01.379 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:01.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.391+0000 7f8be6d75700 1 -- 172.21.15.123:0/2553129032 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8bcc02ff60 msgr2=0x7f8bcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:01.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.391+0000 7f8be6d75700 1 --2- 172.21.15.123:0/2553129032 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8bcc02ff60 0x7f8bcc032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f8bd00096a0 tx=0x7f8bd0006b40).stop 2024-01-13T11:53:01.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.391+0000 7f8be6d75700 1 -- 172.21.15.123:0/2553129032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8be00f51a0 msgr2=0x7f8be01064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:01.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.391+0000 7f8be6d75700 1 --2- 172.21.15.123:0/2553129032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8be00f51a0 0x7f8be01064a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f8bdc013510 tx=0x7f8bdc00b040).stop 2024-01-13T11:53:01.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.391+0000 7f8be6d75700 1 -- 172.21.15.123:0/2553129032 shutdown_connections 2024-01-13T11:53:01.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.391+0000 7f8be6d75700 1 --2- 172.21.15.123:0/2553129032 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8bcc02ff60 0x7f8bcc032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:01.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.391+0000 7f8be6d75700 1 --2- 172.21.15.123:0/2553129032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8be00f51a0 0x7f8be01064a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:01.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.391+0000 7f8be6d75700 1 -- 172.21.15.123:0/2553129032 >> 172.21.15.123:0/2553129032 conn(0x7f8be00f09a0 msgr2=0x7f8be005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:01.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.391+0000 7f8be6d75700 1 -- 172.21.15.123:0/2553129032 shutdown_connections 2024-01-13T11:53:01.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:01.391+0000 7f8be6d75700 1 -- 172.21.15.123:0/2553129032 wait complete. 2024-01-13T11:53:01.381 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:01.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:01 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:52:59.530919+0000 mgr.smithi059.hmyldj (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:02.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:02 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:01.390545+0000 mon.smithi059 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.123:0/2553129032' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:02.781 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:02.782 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:03.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:03 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:01.531196+0000 mgr.smithi059.hmyldj (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:03.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.972+0000 7f267351f700 1 -- 172.21.15.123:0/3702672128 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f266c0f6510 msgr2=0x7f266c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:03.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.972+0000 7f267351f700 1 --2- 172.21.15.123:0/3702672128 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f266c0f6510 0x7f266c0f6930 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f265c009fb0 tx=0x7f265c009b30).stop 2024-01-13T11:53:03.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.972+0000 7f267351f700 1 -- 172.21.15.123:0/3702672128 shutdown_connections 2024-01-13T11:53:03.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.972+0000 7f267351f700 1 --2- 172.21.15.123:0/3702672128 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f266c0f6510 0x7f266c0f6930 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:03.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.972+0000 7f267351f700 1 -- 172.21.15.123:0/3702672128 >> 172.21.15.123:0/3702672128 conn(0x7f266c0f1c70 msgr2=0x7f266c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:03.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.972+0000 7f267351f700 1 -- 172.21.15.123:0/3702672128 shutdown_connections 2024-01-13T11:53:03.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.972+0000 7f267351f700 1 -- 172.21.15.123:0/3702672128 wait complete. 2024-01-13T11:53:03.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.973+0000 7f267351f700 1 Processor -- start 2024-01-13T11:53:03.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.973+0000 7f267351f700 1 -- start start 2024-01-13T11:53:03.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.973+0000 7f267351f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f266c0f6510 0x7f266c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:03.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.973+0000 7f267351f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c106c20 con 0x7f266c0f6510 2024-01-13T11:53:03.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.973+0000 7f26712bb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f266c0f6510 0x7f266c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:03.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.973+0000 7f26712bb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f266c0f6510 0x7f266c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:36696/0 (socket says 172.21.15.123:36696) 2024-01-13T11:53:03.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.973+0000 7f26712bb700 1 -- 172.21.15.123:0/2657700054 learned_addr learned my addr 172.21.15.123:0/2657700054 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:03.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.973+0000 7f26712bb700 1 -- 172.21.15.123:0/2657700054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265c005040 con 0x7f266c0f6510 2024-01-13T11:53:03.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.973+0000 7f26712bb700 1 --2- 172.21.15.123:0/2657700054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f266c0f6510 0x7f266c106730 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f265c013510 tx=0x7f265c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:03.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.974+0000 7f2661ffb700 1 -- 172.21.15.123:0/2657700054 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f265c009320 con 0x7f266c0f6510 2024-01-13T11:53:03.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.974+0000 7f267351f700 1 -- 172.21.15.123:0/2657700054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266c106e20 con 0x7f266c0f6510 2024-01-13T11:53:03.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.974+0000 7f2661ffb700 1 -- 172.21.15.123:0/2657700054 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f265c013f00 con 0x7f266c0f6510 2024-01-13T11:53:03.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.974+0000 7f2661ffb700 1 -- 172.21.15.123:0/2657700054 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f265c014d90 con 0x7f266c0f6510 2024-01-13T11:53:03.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.974+0000 7f267351f700 1 -- 172.21.15.123:0/2657700054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f266c1072b0 con 0x7f266c0f6510 2024-01-13T11:53:03.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.975+0000 7f2661ffb700 1 -- 172.21.15.123:0/2657700054 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f265c0095f0 con 0x7f266c0f6510 2024-01-13T11:53:03.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.975+0000 7f267351f700 1 -- 172.21.15.123:0/2657700054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f266c045ad0 con 0x7f266c0f6510 2024-01-13T11:53:03.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.975+0000 7f2661ffb700 1 --2- 172.21.15.123:0/2657700054 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f265802ff60 0x7f2658032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:03.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.975+0000 7f2661ffb700 1 -- 172.21.15.123:0/2657700054 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f265c040fc0 con 0x7f266c0f6510 2024-01-13T11:53:03.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.975+0000 7f2670aba700 1 --2- 172.21.15.123:0/2657700054 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f265802ff60 0x7f2658032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:03.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.975+0000 7f2670aba700 1 --2- 172.21.15.123:0/2657700054 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f265802ff60 0x7f2658032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f26680096a0 tx=0x7f2668006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:03.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:03.980+0000 7f2661ffb700 1 -- 172.21.15.123:0/2657700054 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f265c014420 con 0x7f266c0f6510 2024-01-13T11:53:04.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:04.183+0000 7f267351f700 1 -- 172.21.15.123:0/2657700054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f266c0fbf60 con 0x7f266c0f6510 2024-01-13T11:53:04.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:04.184+0000 7f2661ffb700 1 -- 172.21.15.123:0/2657700054 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f265c010230 con 0x7f266c0f6510 2024-01-13T11:53:04.173 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:04.173 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:04.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:04.185+0000 7f267351f700 1 -- 172.21.15.123:0/2657700054 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f265802ff60 msgr2=0x7f2658032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:04.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:04.185+0000 7f267351f700 1 --2- 172.21.15.123:0/2657700054 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f265802ff60 0x7f2658032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f26680096a0 tx=0x7f2668006b40).stop 2024-01-13T11:53:04.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:04.185+0000 7f267351f700 1 -- 172.21.15.123:0/2657700054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f266c0f6510 msgr2=0x7f266c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:04.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:04.185+0000 7f267351f700 1 --2- 172.21.15.123:0/2657700054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f266c0f6510 0x7f266c106730 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f265c013510 tx=0x7f265c00b040).stop 2024-01-13T11:53:04.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:04.185+0000 7f267351f700 1 -- 172.21.15.123:0/2657700054 shutdown_connections 2024-01-13T11:53:04.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:04.185+0000 7f267351f700 1 --2- 172.21.15.123:0/2657700054 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f265802ff60 0x7f2658032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:04.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:04.185+0000 7f267351f700 1 --2- 172.21.15.123:0/2657700054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f266c0f6510 0x7f266c106730 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:04.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:04.185+0000 7f267351f700 1 -- 172.21.15.123:0/2657700054 >> 172.21.15.123:0/2657700054 conn(0x7f266c0f1c70 msgr2=0x7f266c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:04.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:04.186+0000 7f267351f700 1 -- 172.21.15.123:0/2657700054 shutdown_connections 2024-01-13T11:53:04.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:04.186+0000 7f267351f700 1 -- 172.21.15.123:0/2657700054 wait complete. 2024-01-13T11:53:04.175 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:04.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:04 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:04.185117+0000 mon.smithi059 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.123:0/2657700054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:05.555 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:05.555 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:05.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:05 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:03.531537+0000 mgr.smithi059.hmyldj (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:06.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.752+0000 7fc1a9097700 1 -- 172.21.15.123:0/2262278807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a40f6510 msgr2=0x7fc1a40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:06.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.752+0000 7fc1a9097700 1 --2- 172.21.15.123:0/2262278807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a40f6510 0x7fc1a40f6930 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fc18c009fb0 tx=0x7fc18c009b30).stop 2024-01-13T11:53:06.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.752+0000 7fc1a9097700 1 -- 172.21.15.123:0/2262278807 shutdown_connections 2024-01-13T11:53:06.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.752+0000 7fc1a9097700 1 --2- 172.21.15.123:0/2262278807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a40f6510 0x7fc1a40f6930 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:06.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.752+0000 7fc1a9097700 1 -- 172.21.15.123:0/2262278807 >> 172.21.15.123:0/2262278807 conn(0x7fc1a40f1c70 msgr2=0x7fc1a40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:06.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.752+0000 7fc1a9097700 1 -- 172.21.15.123:0/2262278807 shutdown_connections 2024-01-13T11:53:06.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.752+0000 7fc1a9097700 1 -- 172.21.15.123:0/2262278807 wait complete. 2024-01-13T11:53:06.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.753+0000 7fc1a9097700 1 Processor -- start 2024-01-13T11:53:06.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.753+0000 7fc1a9097700 1 -- start start 2024-01-13T11:53:06.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.753+0000 7fc1a9097700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a40f6510 0x7fc1a4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:06.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.753+0000 7fc1a9097700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a4106c20 con 0x7fc1a40f6510 2024-01-13T11:53:06.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.753+0000 7fc1a2d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a40f6510 0x7fc1a4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:06.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.753+0000 7fc1a2d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a40f6510 0x7fc1a4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:57112/0 (socket says 172.21.15.123:57112) 2024-01-13T11:53:06.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.753+0000 7fc1a2d9d700 1 -- 172.21.15.123:0/2796404177 learned_addr learned my addr 172.21.15.123:0/2796404177 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:06.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.753+0000 7fc1a2d9d700 1 -- 172.21.15.123:0/2796404177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc18c005040 con 0x7fc1a40f6510 2024-01-13T11:53:06.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.753+0000 7fc1a2d9d700 1 --2- 172.21.15.123:0/2796404177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a40f6510 0x7fc1a4106730 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fc18c013510 tx=0x7fc18c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:06.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.754+0000 7fc19b7fe700 1 -- 172.21.15.123:0/2796404177 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc18c009320 con 0x7fc1a40f6510 2024-01-13T11:53:06.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.754+0000 7fc1a9097700 1 -- 172.21.15.123:0/2796404177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a4106e20 con 0x7fc1a40f6510 2024-01-13T11:53:06.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.754+0000 7fc19b7fe700 1 -- 172.21.15.123:0/2796404177 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc18c013f00 con 0x7fc1a40f6510 2024-01-13T11:53:06.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.754+0000 7fc19b7fe700 1 -- 172.21.15.123:0/2796404177 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc18c014d90 con 0x7fc1a40f6510 2024-01-13T11:53:06.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.754+0000 7fc1a9097700 1 -- 172.21.15.123:0/2796404177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a41072b0 con 0x7fc1a40f6510 2024-01-13T11:53:06.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.754+0000 7fc19b7fe700 1 -- 172.21.15.123:0/2796404177 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc18c0263a0 con 0x7fc1a40f6510 2024-01-13T11:53:06.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.755+0000 7fc19b7fe700 1 --2- 172.21.15.123:0/2796404177 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc19002ff60 0x7fc190032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:06.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.755+0000 7fc19b7fe700 1 -- 172.21.15.123:0/2796404177 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc18c041100 con 0x7fc1a40f6510 2024-01-13T11:53:06.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.755+0000 7fc1a9097700 1 -- 172.21.15.123:0/2796404177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a4045ad0 con 0x7fc1a40f6510 2024-01-13T11:53:06.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.755+0000 7fc1a259c700 1 --2- 172.21.15.123:0/2796404177 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc19002ff60 0x7fc190032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:06.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.755+0000 7fc1a259c700 1 --2- 172.21.15.123:0/2796404177 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc19002ff60 0x7fc190032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fc1940096a0 tx=0x7fc194006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:06.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.760+0000 7fc19b7fe700 1 -- 172.21.15.123:0/2796404177 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc18c012920 con 0x7fc1a40f6510 2024-01-13T11:53:06.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.967+0000 7fc1a9097700 1 -- 172.21.15.123:0/2796404177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1a40fc050 con 0x7fc1a40f6510 2024-01-13T11:53:06.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.967+0000 7fc19b7fe700 1 -- 172.21.15.123:0/2796404177 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc18c02b020 con 0x7fc1a40f6510 2024-01-13T11:53:06.956 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:06.956 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:06.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.969+0000 7fc1a9097700 1 -- 172.21.15.123:0/2796404177 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc19002ff60 msgr2=0x7fc190032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:06.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.969+0000 7fc1a9097700 1 --2- 172.21.15.123:0/2796404177 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc19002ff60 0x7fc190032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fc1940096a0 tx=0x7fc194006b40).stop 2024-01-13T11:53:06.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.969+0000 7fc1a9097700 1 -- 172.21.15.123:0/2796404177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a40f6510 msgr2=0x7fc1a4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:06.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.969+0000 7fc1a9097700 1 --2- 172.21.15.123:0/2796404177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a40f6510 0x7fc1a4106730 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fc18c013510 tx=0x7fc18c00b040).stop 2024-01-13T11:53:06.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.969+0000 7fc1a9097700 1 -- 172.21.15.123:0/2796404177 shutdown_connections 2024-01-13T11:53:06.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.969+0000 7fc1a9097700 1 --2- 172.21.15.123:0/2796404177 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc19002ff60 0x7fc190032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:06.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.969+0000 7fc1a9097700 1 --2- 172.21.15.123:0/2796404177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a40f6510 0x7fc1a4106730 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:06.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.969+0000 7fc1a9097700 1 -- 172.21.15.123:0/2796404177 >> 172.21.15.123:0/2796404177 conn(0x7fc1a40f1c70 msgr2=0x7fc1a40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:06.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.969+0000 7fc1a9097700 1 -- 172.21.15.123:0/2796404177 shutdown_connections 2024-01-13T11:53:06.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:06.969+0000 7fc1a9097700 1 -- 172.21.15.123:0/2796404177 wait complete. 2024-01-13T11:53:06.959 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:07.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:05.531913+0000 mgr.smithi059.hmyldj (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:07.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:06.968511+0000 mon.smithi059 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.123:0/2796404177' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:08.349 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:08.349 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:08.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:08 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:07.532217+0000 mgr.smithi059.hmyldj (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:09.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.454+0000 7fa879594700 1 -- 172.21.15.123:0/1536566732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740f4740 msgr2=0x7fa8740f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:09.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.454+0000 7fa879594700 1 --2- 172.21.15.123:0/1536566732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740f4740 0x7fa8740f4b60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fa868009fb0 tx=0x7fa868009b30).stop 2024-01-13T11:53:09.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.454+0000 7fa879594700 1 -- 172.21.15.123:0/1536566732 shutdown_connections 2024-01-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.454+0000 7fa879594700 1 --2- 172.21.15.123:0/1536566732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740f4740 0x7fa8740f4b60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.454+0000 7fa879594700 1 -- 172.21.15.123:0/1536566732 >> 172.21.15.123:0/1536566732 conn(0x7fa8740f0260 msgr2=0x7fa8740f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.454+0000 7fa879594700 1 -- 172.21.15.123:0/1536566732 shutdown_connections 2024-01-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.454+0000 7fa879594700 1 -- 172.21.15.123:0/1536566732 wait complete. 2024-01-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.455+0000 7fa879594700 1 Processor -- start 2024-01-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.455+0000 7fa879594700 1 -- start start 2024-01-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.455+0000 7fa879594700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740f4740 0x7fa8741066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.455+0000 7fa879594700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa874106bc0 con 0x7fa8740f4740 2024-01-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.456+0000 7fa872ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740f4740 0x7fa8741066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.456+0000 7fa872ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740f4740 0x7fa8741066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:57134/0 (socket says 172.21.15.123:57134) 2024-01-13T11:53:09.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.456+0000 7fa872ffd700 1 -- 172.21.15.123:0/1683970972 learned_addr learned my addr 172.21.15.123:0/1683970972 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.456+0000 7fa872ffd700 1 -- 172.21.15.123:0/1683970972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa868005040 con 0x7fa8740f4740 2024-01-13T11:53:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.456+0000 7fa872ffd700 1 --2- 172.21.15.123:0/1683970972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740f4740 0x7fa8741066d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fa868013510 tx=0x7fa86800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.456+0000 7fa85b7fe700 1 -- 172.21.15.123:0/1683970972 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa868009320 con 0x7fa8740f4740 2024-01-13T11:53:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.456+0000 7fa879594700 1 -- 172.21.15.123:0/1683970972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa874106dc0 con 0x7fa8740f4740 2024-01-13T11:53:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.456+0000 7fa85b7fe700 1 -- 172.21.15.123:0/1683970972 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa868013f00 con 0x7fa8740f4740 2024-01-13T11:53:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.456+0000 7fa85b7fe700 1 -- 172.21.15.123:0/1683970972 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa868014d90 con 0x7fa8740f4740 2024-01-13T11:53:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.456+0000 7fa879594700 1 -- 172.21.15.123:0/1683970972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa874107250 con 0x7fa8740f4740 2024-01-13T11:53:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.457+0000 7fa85b7fe700 1 -- 172.21.15.123:0/1683970972 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8680263a0 con 0x7fa8740f4740 2024-01-13T11:53:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.457+0000 7fa85b7fe700 1 --2- 172.21.15.123:0/1683970972 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa85c02ffb0 0x7fa85c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.457+0000 7fa85b7fe700 1 -- 172.21.15.123:0/1683970972 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8680095f0 con 0x7fa8740f4740 2024-01-13T11:53:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.457+0000 7fa879594700 1 -- 172.21.15.123:0/1683970972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa874045ad0 con 0x7fa8740f4740 2024-01-13T11:53:09.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.458+0000 7fa8727fc700 1 --2- 172.21.15.123:0/1683970972 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa85c02ffb0 0x7fa85c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:09.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.458+0000 7fa8727fc700 1 --2- 172.21.15.123:0/1683970972 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa85c02ffb0 0x7fa85c032470 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fa8640096a0 tx=0x7fa864006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:09.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.462+0000 7fa85b7fe700 1 -- 172.21.15.123:0/1683970972 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa868012920 con 0x7fa8740f4740 2024-01-13T11:53:09.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.667+0000 7fa879594700 1 -- 172.21.15.123:0/1683970972 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8740fbf40 con 0x7fa8740f4740 2024-01-13T11:53:09.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.668+0000 7fa85b7fe700 1 -- 172.21.15.123:0/1683970972 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa86802b020 con 0x7fa8740f4740 2024-01-13T11:53:09.657 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:09.657 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:09.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.670+0000 7fa879594700 1 -- 172.21.15.123:0/1683970972 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa85c02ffb0 msgr2=0x7fa85c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:09.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.670+0000 7fa879594700 1 --2- 172.21.15.123:0/1683970972 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa85c02ffb0 0x7fa85c032470 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fa8640096a0 tx=0x7fa864006b40).stop 2024-01-13T11:53:09.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.670+0000 7fa879594700 1 -- 172.21.15.123:0/1683970972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740f4740 msgr2=0x7fa8741066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:09.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.670+0000 7fa879594700 1 --2- 172.21.15.123:0/1683970972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740f4740 0x7fa8741066d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fa868013510 tx=0x7fa86800b040).stop 2024-01-13T11:53:09.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.670+0000 7fa879594700 1 -- 172.21.15.123:0/1683970972 shutdown_connections 2024-01-13T11:53:09.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.670+0000 7fa879594700 1 --2- 172.21.15.123:0/1683970972 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa85c02ffb0 0x7fa85c032470 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:09.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.670+0000 7fa879594700 1 --2- 172.21.15.123:0/1683970972 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740f4740 0x7fa8741066d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:09.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.670+0000 7fa879594700 1 -- 172.21.15.123:0/1683970972 >> 172.21.15.123:0/1683970972 conn(0x7fa8740f0260 msgr2=0x7fa8740f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:09.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.670+0000 7fa879594700 1 -- 172.21.15.123:0/1683970972 shutdown_connections 2024-01-13T11:53:09.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:09.670+0000 7fa879594700 1 -- 172.21.15.123:0/1683970972 wait complete. 2024-01-13T11:53:09.660 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:10.887 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:10 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:09.532600+0000 mgr.smithi059.hmyldj (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:10.888 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:10 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:09.669189+0000 mon.smithi059 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.123:0/1683970972' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:11.085 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:11.085 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.248+0000 7fb1d25b4700 1 -- 172.21.15.123:0/2304314320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1cc060280 msgr2=0x7fb1cc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.248+0000 7fb1d25b4700 1 --2- 172.21.15.123:0/2304314320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1cc060280 0x7fb1cc0f90f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fb1c0009fb0 tx=0x7fb1c0009b30).stop 2024-01-13T11:53:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.248+0000 7fb1d25b4700 1 -- 172.21.15.123:0/2304314320 shutdown_connections 2024-01-13T11:53:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.248+0000 7fb1d25b4700 1 --2- 172.21.15.123:0/2304314320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1cc060280 0x7fb1cc0f90f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.248+0000 7fb1d25b4700 1 -- 172.21.15.123:0/2304314320 >> 172.21.15.123:0/2304314320 conn(0x7fb1cc0f0b70 msgr2=0x7fb1cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.248+0000 7fb1d25b4700 1 -- 172.21.15.123:0/2304314320 shutdown_connections 2024-01-13T11:53:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.248+0000 7fb1d25b4700 1 -- 172.21.15.123:0/2304314320 wait complete. 2024-01-13T11:53:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.249+0000 7fb1d25b4700 1 Processor -- start 2024-01-13T11:53:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.249+0000 7fb1d25b4700 1 -- start start 2024-01-13T11:53:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.249+0000 7fb1d25b4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1cc060280 0x7fb1cc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:12.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.249+0000 7fb1d25b4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc106bd0 con 0x7fb1cc060280 2024-01-13T11:53:12.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.249+0000 7fb1cbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1cc060280 0x7fb1cc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:12.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.249+0000 7fb1cbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1cc060280 0x7fb1cc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:57154/0 (socket says 172.21.15.123:57154) 2024-01-13T11:53:12.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.249+0000 7fb1cbfff700 1 -- 172.21.15.123:0/427559392 learned_addr learned my addr 172.21.15.123:0/427559392 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:12.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.249+0000 7fb1cbfff700 1 -- 172.21.15.123:0/427559392 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c0005040 con 0x7fb1cc060280 2024-01-13T11:53:12.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.249+0000 7fb1cbfff700 1 --2- 172.21.15.123:0/427559392 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1cc060280 0x7fb1cc1066e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fb1c0013510 tx=0x7fb1c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:12.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.250+0000 7fb1c8ff9700 1 -- 172.21.15.123:0/427559392 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1c0009320 con 0x7fb1cc060280 2024-01-13T11:53:12.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.250+0000 7fb1d25b4700 1 -- 172.21.15.123:0/427559392 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1cc106dd0 con 0x7fb1cc060280 2024-01-13T11:53:12.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.250+0000 7fb1c8ff9700 1 -- 172.21.15.123:0/427559392 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb1c0013f00 con 0x7fb1cc060280 2024-01-13T11:53:12.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.250+0000 7fb1c8ff9700 1 -- 172.21.15.123:0/427559392 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1c0014dc0 con 0x7fb1cc060280 2024-01-13T11:53:12.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.250+0000 7fb1d25b4700 1 -- 172.21.15.123:0/427559392 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1cc107260 con 0x7fb1cc060280 2024-01-13T11:53:12.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.250+0000 7fb1c8ff9700 1 -- 172.21.15.123:0/427559392 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1c00095f0 con 0x7fb1cc060280 2024-01-13T11:53:12.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.251+0000 7fb1d25b4700 1 -- 172.21.15.123:0/427559392 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1cc045ad0 con 0x7fb1cc060280 2024-01-13T11:53:12.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.251+0000 7fb1c8ff9700 1 --2- 172.21.15.123:0/427559392 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb1b402ffb0 0x7fb1b4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:12.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.251+0000 7fb1c8ff9700 1 -- 172.21.15.123:0/427559392 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb1c0041100 con 0x7fb1cc060280 2024-01-13T11:53:12.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.251+0000 7fb1cb7fe700 1 --2- 172.21.15.123:0/427559392 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb1b402ffb0 0x7fb1b4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:12.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.251+0000 7fb1cb7fe700 1 --2- 172.21.15.123:0/427559392 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb1b402ffb0 0x7fb1b4032470 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fb1bc0096a0 tx=0x7fb1bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:12.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.256+0000 7fb1c8ff9700 1 -- 172.21.15.123:0/427559392 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb1c0014420 con 0x7fb1cc060280 2024-01-13T11:53:12.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.461+0000 7fb1d25b4700 1 -- 172.21.15.123:0/427559392 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1cc108110 con 0x7fb1cc060280 2024-01-13T11:53:12.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.461+0000 7fb1c8ff9700 1 -- 172.21.15.123:0/427559392 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb1c0010230 con 0x7fb1cc060280 2024-01-13T11:53:12.450 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:12.450 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.463+0000 7fb1d25b4700 1 -- 172.21.15.123:0/427559392 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb1b402ffb0 msgr2=0x7fb1b4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.463+0000 7fb1d25b4700 1 --2- 172.21.15.123:0/427559392 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb1b402ffb0 0x7fb1b4032470 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fb1bc0096a0 tx=0x7fb1bc006b40).stop 2024-01-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.463+0000 7fb1d25b4700 1 -- 172.21.15.123:0/427559392 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1cc060280 msgr2=0x7fb1cc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.463+0000 7fb1d25b4700 1 --2- 172.21.15.123:0/427559392 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1cc060280 0x7fb1cc1066e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fb1c0013510 tx=0x7fb1c000b040).stop 2024-01-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.463+0000 7fb1d25b4700 1 -- 172.21.15.123:0/427559392 shutdown_connections 2024-01-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.463+0000 7fb1d25b4700 1 --2- 172.21.15.123:0/427559392 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb1b402ffb0 0x7fb1b4032470 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:12.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.463+0000 7fb1d25b4700 1 --2- 172.21.15.123:0/427559392 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1cc060280 0x7fb1cc1066e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:12.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.463+0000 7fb1d25b4700 1 -- 172.21.15.123:0/427559392 >> 172.21.15.123:0/427559392 conn(0x7fb1cc0f0b70 msgr2=0x7fb1cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:12.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.463+0000 7fb1d25b4700 1 -- 172.21.15.123:0/427559392 shutdown_connections 2024-01-13T11:53:12.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:12.463+0000 7fb1d25b4700 1 -- 172.21.15.123:0/427559392 wait complete. 2024-01-13T11:53:12.453 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:13.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:13 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:11.532847+0000 mgr.smithi059.hmyldj (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:13.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:13 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:12.462714+0000 mon.smithi059 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.123:0/427559392' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:13.832 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:13.833 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:14.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.954+0000 7f37c6baa700 1 -- 172.21.15.123:0/1694396531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c00f5410 msgr2=0x7f37c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:14.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.954+0000 7f37c6baa700 1 --2- 172.21.15.123:0/1694396531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c00f5410 0x7f37c00f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f37b4009fb0 tx=0x7f37b4009b30).stop 2024-01-13T11:53:14.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.954+0000 7f37c6baa700 1 -- 172.21.15.123:0/1694396531 shutdown_connections 2024-01-13T11:53:14.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.954+0000 7f37c6baa700 1 --2- 172.21.15.123:0/1694396531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c00f5410 0x7f37c00f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:14.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.954+0000 7f37c6baa700 1 -- 172.21.15.123:0/1694396531 >> 172.21.15.123:0/1694396531 conn(0x7f37c00f0b70 msgr2=0x7f37c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:14.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.954+0000 7f37c6baa700 1 -- 172.21.15.123:0/1694396531 shutdown_connections 2024-01-13T11:53:14.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.954+0000 7f37c6baa700 1 -- 172.21.15.123:0/1694396531 wait complete. 2024-01-13T11:53:14.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.955+0000 7f37c6baa700 1 Processor -- start 2024-01-13T11:53:14.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.955+0000 7f37c6baa700 1 -- start start 2024-01-13T11:53:14.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.955+0000 7f37c6baa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c00f5410 0x7f37c0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:14.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.955+0000 7f37c6baa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c0105b20 con 0x7f37c00f5410 2024-01-13T11:53:14.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.955+0000 7f37c4946700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c00f5410 0x7f37c0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:14.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.955+0000 7f37c4946700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c00f5410 0x7f37c0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:57184/0 (socket says 172.21.15.123:57184) 2024-01-13T11:53:14.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.955+0000 7f37c4946700 1 -- 172.21.15.123:0/2622487630 learned_addr learned my addr 172.21.15.123:0/2622487630 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:14.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.956+0000 7f37c4946700 1 -- 172.21.15.123:0/2622487630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b4005040 con 0x7f37c00f5410 2024-01-13T11:53:14.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.956+0000 7f37c4946700 1 --2- 172.21.15.123:0/2622487630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c00f5410 0x7f37c0105630 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f37b4013510 tx=0x7f37b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:14.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.956+0000 7f37bd7fa700 1 -- 172.21.15.123:0/2622487630 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37b4009320 con 0x7f37c00f5410 2024-01-13T11:53:14.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.956+0000 7f37c6baa700 1 -- 172.21.15.123:0/2622487630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c0105d20 con 0x7f37c00f5410 2024-01-13T11:53:14.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.956+0000 7f37bd7fa700 1 -- 172.21.15.123:0/2622487630 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37b4013f00 con 0x7f37c00f5410 2024-01-13T11:53:14.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.956+0000 7f37bd7fa700 1 -- 172.21.15.123:0/2622487630 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37b4014d90 con 0x7f37c00f5410 2024-01-13T11:53:14.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.956+0000 7f37c6baa700 1 -- 172.21.15.123:0/2622487630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c01061b0 con 0x7f37c00f5410 2024-01-13T11:53:14.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.957+0000 7f37bd7fa700 1 -- 172.21.15.123:0/2622487630 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37b4014420 con 0x7f37c00f5410 2024-01-13T11:53:14.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.957+0000 7f37c6baa700 1 -- 172.21.15.123:0/2622487630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c0045ad0 con 0x7f37c00f5410 2024-01-13T11:53:14.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.957+0000 7f37bd7fa700 1 --2- 172.21.15.123:0/2622487630 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f37b002ff60 0x7f37b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:14.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.957+0000 7f37bd7fa700 1 -- 172.21.15.123:0/2622487630 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f37b40410c0 con 0x7f37c00f5410 2024-01-13T11:53:14.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.957+0000 7f37bffff700 1 --2- 172.21.15.123:0/2622487630 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f37b002ff60 0x7f37b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:14.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.957+0000 7f37bffff700 1 --2- 172.21.15.123:0/2622487630 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f37b002ff60 0x7f37b0032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f37ac0096a0 tx=0x7f37ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:14.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:14.962+0000 7f37bd7fa700 1 -- 172.21.15.123:0/2622487630 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37b40095f0 con 0x7f37c00f5410 2024-01-13T11:53:15.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:15.169+0000 7f37c6baa700 1 -- 172.21.15.123:0/2622487630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37c00fae60 con 0x7f37c00f5410 2024-01-13T11:53:15.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:15.169+0000 7f37bd7fa700 1 -- 172.21.15.123:0/2622487630 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f37b401db70 con 0x7f37c00f5410 2024-01-13T11:53:15.157 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:15.157 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:15.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:15.171+0000 7f37c6baa700 1 -- 172.21.15.123:0/2622487630 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f37b002ff60 msgr2=0x7f37b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:15.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:15.171+0000 7f37c6baa700 1 --2- 172.21.15.123:0/2622487630 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f37b002ff60 0x7f37b0032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f37ac0096a0 tx=0x7f37ac006b40).stop 2024-01-13T11:53:15.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:15.171+0000 7f37c6baa700 1 -- 172.21.15.123:0/2622487630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c00f5410 msgr2=0x7f37c0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:15.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:15.171+0000 7f37c6baa700 1 --2- 172.21.15.123:0/2622487630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c00f5410 0x7f37c0105630 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f37b4013510 tx=0x7f37b400b040).stop 2024-01-13T11:53:15.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:15.171+0000 7f37c6baa700 1 -- 172.21.15.123:0/2622487630 shutdown_connections 2024-01-13T11:53:15.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:15.171+0000 7f37c6baa700 1 --2- 172.21.15.123:0/2622487630 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f37b002ff60 0x7f37b0032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:15.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:15.171+0000 7f37c6baa700 1 --2- 172.21.15.123:0/2622487630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37c00f5410 0x7f37c0105630 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:15.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:15.171+0000 7f37c6baa700 1 -- 172.21.15.123:0/2622487630 >> 172.21.15.123:0/2622487630 conn(0x7f37c00f0b70 msgr2=0x7f37c00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:15.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:15.171+0000 7f37c6baa700 1 -- 172.21.15.123:0/2622487630 shutdown_connections 2024-01-13T11:53:15.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:15.171+0000 7f37c6baa700 1 -- 172.21.15.123:0/2622487630 wait complete. 2024-01-13T11:53:15.160 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:15.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:15 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:13.533110+0000 mgr.smithi059.hmyldj (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:15.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:15 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:15.169628+0000 mon.smithi059 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.123:0/2622487630' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:16.538 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:16.538 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:17.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:17 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:15.533344+0000 mgr.smithi059.hmyldj (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:17.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.691+0000 7f0ff161f700 1 -- 172.21.15.123:0/995251604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fec0f47c0 msgr2=0x7f0fec0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:17.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.691+0000 7f0ff161f700 1 --2- 172.21.15.123:0/995251604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fec0f47c0 0x7f0fec0f4be0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f0fdc009fb0 tx=0x7f0fdc009b30).stop 2024-01-13T11:53:17.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.691+0000 7f0ff161f700 1 -- 172.21.15.123:0/995251604 shutdown_connections 2024-01-13T11:53:17.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.691+0000 7f0ff161f700 1 --2- 172.21.15.123:0/995251604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fec0f47c0 0x7f0fec0f4be0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:17.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.691+0000 7f0ff161f700 1 -- 172.21.15.123:0/995251604 >> 172.21.15.123:0/995251604 conn(0x7f0fec0f02e0 msgr2=0x7f0fec0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:17.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.691+0000 7f0ff161f700 1 -- 172.21.15.123:0/995251604 shutdown_connections 2024-01-13T11:53:17.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.691+0000 7f0ff161f700 1 -- 172.21.15.123:0/995251604 wait complete. 2024-01-13T11:53:17.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.691+0000 7f0ff161f700 1 Processor -- start 2024-01-13T11:53:17.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.691+0000 7f0ff161f700 1 -- start start 2024-01-13T11:53:17.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.692+0000 7f0ff161f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fec0f47c0 0x7f0fec0f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:17.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.692+0000 7f0ff161f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec0f57f0 con 0x7f0fec0f47c0 2024-01-13T11:53:17.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.692+0000 7f0feaffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fec0f47c0 0x7f0fec0f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:17.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.692+0000 7f0feaffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fec0f47c0 0x7f0fec0f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:51330/0 (socket says 172.21.15.123:51330) 2024-01-13T11:53:17.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.692+0000 7f0feaffd700 1 -- 172.21.15.123:0/2469473696 learned_addr learned my addr 172.21.15.123:0/2469473696 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:17.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.692+0000 7f0feaffd700 1 -- 172.21.15.123:0/2469473696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fdc005040 con 0x7f0fec0f47c0 2024-01-13T11:53:17.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.692+0000 7f0feaffd700 1 --2- 172.21.15.123:0/2469473696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fec0f47c0 0x7f0fec0f7370 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f0fdc014000 tx=0x7f0fdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:17.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.692+0000 7f0fd37fe700 1 -- 172.21.15.123:0/2469473696 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fdc0095f0 con 0x7f0fec0f47c0 2024-01-13T11:53:17.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.692+0000 7f0ff161f700 1 -- 172.21.15.123:0/2469473696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fec0f5990 con 0x7f0fec0f47c0 2024-01-13T11:53:17.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.692+0000 7f0fd37fe700 1 -- 172.21.15.123:0/2469473696 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0fdc007e70 con 0x7f0fec0f47c0 2024-01-13T11:53:17.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.692+0000 7f0fd37fe700 1 -- 172.21.15.123:0/2469473696 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fdc014ea0 con 0x7f0fec0f47c0 2024-01-13T11:53:17.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.693+0000 7f0ff161f700 1 -- 172.21.15.123:0/2469473696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fec0f5e20 con 0x7f0fec0f47c0 2024-01-13T11:53:17.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.693+0000 7f0fd37fe700 1 -- 172.21.15.123:0/2469473696 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0fdc014690 con 0x7f0fec0f47c0 2024-01-13T11:53:17.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.693+0000 7f0ff161f700 1 -- 172.21.15.123:0/2469473696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fec059100 con 0x7f0fec0f47c0 2024-01-13T11:53:17.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.694+0000 7f0fd37fe700 1 --2- 172.21.15.123:0/2469473696 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0fd402ffb0 0x7f0fd4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:17.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.694+0000 7f0fd37fe700 1 -- 172.21.15.123:0/2469473696 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0fdc040930 con 0x7f0fec0f47c0 2024-01-13T11:53:17.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.694+0000 7f0fea7fc700 1 --2- 172.21.15.123:0/2469473696 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0fd402ffb0 0x7f0fd4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:17.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.694+0000 7f0fea7fc700 1 --2- 172.21.15.123:0/2469473696 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0fd402ffb0 0x7f0fd4032470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f0fe00096a0 tx=0x7f0fe0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:17.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.699+0000 7f0fd37fe700 1 -- 172.21.15.123:0/2469473696 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0fdc00f300 con 0x7f0fec0f47c0 2024-01-13T11:53:17.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.906+0000 7f0ff161f700 1 -- 172.21.15.123:0/2469473696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0fec0fbf60 con 0x7f0fec0f47c0 2024-01-13T11:53:17.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.907+0000 7f0fd37fe700 1 -- 172.21.15.123:0/2469473696 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0fdc00f300 con 0x7f0fec0f47c0 2024-01-13T11:53:17.895 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:17.895 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:17.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.909+0000 7f0ff161f700 1 -- 172.21.15.123:0/2469473696 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0fd402ffb0 msgr2=0x7f0fd4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:17.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.909+0000 7f0ff161f700 1 --2- 172.21.15.123:0/2469473696 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0fd402ffb0 0x7f0fd4032470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f0fe00096a0 tx=0x7f0fe0006b40).stop 2024-01-13T11:53:17.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.909+0000 7f0ff161f700 1 -- 172.21.15.123:0/2469473696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fec0f47c0 msgr2=0x7f0fec0f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:17.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.909+0000 7f0ff161f700 1 --2- 172.21.15.123:0/2469473696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fec0f47c0 0x7f0fec0f7370 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f0fdc014000 tx=0x7f0fdc00b040).stop 2024-01-13T11:53:17.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.909+0000 7f0ff161f700 1 -- 172.21.15.123:0/2469473696 shutdown_connections 2024-01-13T11:53:17.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.909+0000 7f0ff161f700 1 --2- 172.21.15.123:0/2469473696 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0fd402ffb0 0x7f0fd4032470 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:17.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.909+0000 7f0ff161f700 1 --2- 172.21.15.123:0/2469473696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fec0f47c0 0x7f0fec0f7370 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:17.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.909+0000 7f0ff161f700 1 -- 172.21.15.123:0/2469473696 >> 172.21.15.123:0/2469473696 conn(0x7f0fec0f02e0 msgr2=0x7f0fec0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:17.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.909+0000 7f0ff161f700 1 -- 172.21.15.123:0/2469473696 shutdown_connections 2024-01-13T11:53:17.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:17.909+0000 7f0ff161f700 1 -- 172.21.15.123:0/2469473696 wait complete. 2024-01-13T11:53:17.898 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:18.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:18 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:17.907479+0000 mon.smithi059 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.123:0/2469473696' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:19.324 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:19.324 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:19.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:19 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:17.533628+0000 mgr.smithi059.hmyldj (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:20.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.438+0000 7f07c3731700 1 -- 172.21.15.123:0/697664719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07bc0f5410 msgr2=0x7f07bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:20.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.438+0000 7f07c3731700 1 --2- 172.21.15.123:0/697664719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07bc0f5410 0x7f07bc0f5830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f07ac009fb0 tx=0x7f07ac009b30).stop 2024-01-13T11:53:20.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.438+0000 7f07c3731700 1 -- 172.21.15.123:0/697664719 shutdown_connections 2024-01-13T11:53:20.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.438+0000 7f07c3731700 1 --2- 172.21.15.123:0/697664719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07bc0f5410 0x7f07bc0f5830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:20.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.438+0000 7f07c3731700 1 -- 172.21.15.123:0/697664719 >> 172.21.15.123:0/697664719 conn(0x7f07bc0f0b70 msgr2=0x7f07bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:20.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.438+0000 7f07c3731700 1 -- 172.21.15.123:0/697664719 shutdown_connections 2024-01-13T11:53:20.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.438+0000 7f07c3731700 1 -- 172.21.15.123:0/697664719 wait complete. 2024-01-13T11:53:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.439+0000 7f07c3731700 1 Processor -- start 2024-01-13T11:53:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.439+0000 7f07c3731700 1 -- start start 2024-01-13T11:53:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.439+0000 7f07c3731700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07bc0f5410 0x7f07bc10b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.439+0000 7f07c3731700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc107240 con 0x7f07bc0f5410 2024-01-13T11:53:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.439+0000 7f07c14cd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07bc0f5410 0x7f07bc10b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.439+0000 7f07c14cd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07bc0f5410 0x7f07bc10b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:51342/0 (socket says 172.21.15.123:51342) 2024-01-13T11:53:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.439+0000 7f07c14cd700 1 -- 172.21.15.123:0/1122760305 learned_addr learned my addr 172.21.15.123:0/1122760305 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.440+0000 7f07c14cd700 1 -- 172.21.15.123:0/1122760305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac005040 con 0x7f07bc0f5410 2024-01-13T11:53:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.440+0000 7f07c14cd700 1 --2- 172.21.15.123:0/1122760305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07bc0f5410 0x7f07bc10b850 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f07ac013510 tx=0x7f07ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.440+0000 7f07b1ffb700 1 -- 172.21.15.123:0/1122760305 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07ac009320 con 0x7f07bc0f5410 2024-01-13T11:53:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.440+0000 7f07c3731700 1 -- 172.21.15.123:0/1122760305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07bc060220 con 0x7f07bc0f5410 2024-01-13T11:53:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.440+0000 7f07b1ffb700 1 -- 172.21.15.123:0/1122760305 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07ac013f00 con 0x7f07bc0f5410 2024-01-13T11:53:20.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.440+0000 7f07b1ffb700 1 -- 172.21.15.123:0/1122760305 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07ac014d90 con 0x7f07bc0f5410 2024-01-13T11:53:20.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.440+0000 7f07c3731700 1 -- 172.21.15.123:0/1122760305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07bc10bd40 con 0x7f07bc0f5410 2024-01-13T11:53:20.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.441+0000 7f07b1ffb700 1 -- 172.21.15.123:0/1122760305 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07ac0095f0 con 0x7f07bc0f5410 2024-01-13T11:53:20.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.441+0000 7f07c3731700 1 -- 172.21.15.123:0/1122760305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07bc045ad0 con 0x7f07bc0f5410 2024-01-13T11:53:20.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.441+0000 7f07b1ffb700 1 --2- 172.21.15.123:0/1122760305 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f07a802ffb0 0x7f07a8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:20.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.441+0000 7f07b1ffb700 1 -- 172.21.15.123:0/1122760305 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07ac0410c0 con 0x7f07bc0f5410 2024-01-13T11:53:20.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.441+0000 7f07c0ccc700 1 --2- 172.21.15.123:0/1122760305 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f07a802ffb0 0x7f07a8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:20.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.442+0000 7f07c0ccc700 1 --2- 172.21.15.123:0/1122760305 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f07a802ffb0 0x7f07a8032470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f07b80096a0 tx=0x7f07b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:20.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.446+0000 7f07b1ffb700 1 -- 172.21.15.123:0/1122760305 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07ac01d7f0 con 0x7f07bc0f5410 2024-01-13T11:53:20.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.654+0000 7f07c3731700 1 -- 172.21.15.123:0/1122760305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07bc0fae60 con 0x7f07bc0f5410 2024-01-13T11:53:20.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.654+0000 7f07b1ffb700 1 -- 172.21.15.123:0/1122760305 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f07ac010230 con 0x7f07bc0f5410 2024-01-13T11:53:20.643 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:20.643 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:20.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.656+0000 7f07c3731700 1 -- 172.21.15.123:0/1122760305 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f07a802ffb0 msgr2=0x7f07a8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:20.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.656+0000 7f07c3731700 1 --2- 172.21.15.123:0/1122760305 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f07a802ffb0 0x7f07a8032470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f07b80096a0 tx=0x7f07b8006b40).stop 2024-01-13T11:53:20.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.656+0000 7f07c3731700 1 -- 172.21.15.123:0/1122760305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07bc0f5410 msgr2=0x7f07bc10b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:20.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.656+0000 7f07c3731700 1 --2- 172.21.15.123:0/1122760305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07bc0f5410 0x7f07bc10b850 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f07ac013510 tx=0x7f07ac00b040).stop 2024-01-13T11:53:20.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.656+0000 7f07c3731700 1 -- 172.21.15.123:0/1122760305 shutdown_connections 2024-01-13T11:53:20.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.656+0000 7f07c3731700 1 --2- 172.21.15.123:0/1122760305 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f07a802ffb0 0x7f07a8032470 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:20.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.656+0000 7f07c3731700 1 --2- 172.21.15.123:0/1122760305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07bc0f5410 0x7f07bc10b850 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:20.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.656+0000 7f07c3731700 1 -- 172.21.15.123:0/1122760305 >> 172.21.15.123:0/1122760305 conn(0x7f07bc0f0b70 msgr2=0x7f07bc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:20.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.656+0000 7f07c3731700 1 -- 172.21.15.123:0/1122760305 shutdown_connections 2024-01-13T11:53:20.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:20.656+0000 7f07c3731700 1 -- 172.21.15.123:0/1122760305 wait complete. 2024-01-13T11:53:20.646 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:21.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:21 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:19.533900+0000 mgr.smithi059.hmyldj (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:21.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:21 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:20.655664+0000 mon.smithi059 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.123:0/1122760305' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:22.034 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:22.035 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:23.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.162+0000 7f150b59e700 1 -- 172.21.15.123:0/2030928003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f150c0601e0 msgr2=0x7f150c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:23.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.162+0000 7f150b59e700 1 --2- 172.21.15.123:0/2030928003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f150c0601e0 0x7f150c0f8f20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f14fc009fb0 tx=0x7f14fc009b30).stop 2024-01-13T11:53:23.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.162+0000 7f150b59e700 1 -- 172.21.15.123:0/2030928003 shutdown_connections 2024-01-13T11:53:23.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.162+0000 7f150b59e700 1 --2- 172.21.15.123:0/2030928003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f150c0601e0 0x7f150c0f8f20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:23.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.162+0000 7f150b59e700 1 -- 172.21.15.123:0/2030928003 >> 172.21.15.123:0/2030928003 conn(0x7f150c0f09a0 msgr2=0x7f150c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:23.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.162+0000 7f150b59e700 1 -- 172.21.15.123:0/2030928003 shutdown_connections 2024-01-13T11:53:23.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.162+0000 7f150b59e700 1 -- 172.21.15.123:0/2030928003 wait complete. 2024-01-13T11:53:23.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.162+0000 7f150b59e700 1 Processor -- start 2024-01-13T11:53:23.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.162+0000 7f150b59e700 1 -- start start 2024-01-13T11:53:23.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.163+0000 7f150b59e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f150c0601e0 0x7f150c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:23.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.163+0000 7f150b59e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150c1069f0 con 0x7f150c0601e0 2024-01-13T11:53:23.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.163+0000 7f150a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f150c0601e0 0x7f150c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:23.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.163+0000 7f150a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f150c0601e0 0x7f150c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:51360/0 (socket says 172.21.15.123:51360) 2024-01-13T11:53:23.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.163+0000 7f150a59c700 1 -- 172.21.15.123:0/1085933503 learned_addr learned my addr 172.21.15.123:0/1085933503 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:23.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.163+0000 7f150a59c700 1 -- 172.21.15.123:0/1085933503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14fc005040 con 0x7f150c0601e0 2024-01-13T11:53:23.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.163+0000 7f150a59c700 1 --2- 172.21.15.123:0/1085933503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f150c0601e0 0x7f150c106500 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f14fc013510 tx=0x7f14fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:23.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.163+0000 7f14faffd700 1 -- 172.21.15.123:0/1085933503 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14fc009320 con 0x7f150c0601e0 2024-01-13T11:53:23.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.163+0000 7f150b59e700 1 -- 172.21.15.123:0/1085933503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f150c106bf0 con 0x7f150c0601e0 2024-01-13T11:53:23.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.163+0000 7f14faffd700 1 -- 172.21.15.123:0/1085933503 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14fc013f00 con 0x7f150c0601e0 2024-01-13T11:53:23.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.163+0000 7f14faffd700 1 -- 172.21.15.123:0/1085933503 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14fc014d90 con 0x7f150c0601e0 2024-01-13T11:53:23.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.163+0000 7f150b59e700 1 -- 172.21.15.123:0/1085933503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f150c107080 con 0x7f150c0601e0 2024-01-13T11:53:23.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.164+0000 7f14faffd700 1 -- 172.21.15.123:0/1085933503 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14fc0095f0 con 0x7f150c0601e0 2024-01-13T11:53:23.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.164+0000 7f150b59e700 1 -- 172.21.15.123:0/1085933503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f150c045ad0 con 0x7f150c0601e0 2024-01-13T11:53:23.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.164+0000 7f14faffd700 1 --2- 172.21.15.123:0/1085933503 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f14f402ff60 0x7f14f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:23.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.164+0000 7f14faffd700 1 -- 172.21.15.123:0/1085933503 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f14fc041000 con 0x7f150c0601e0 2024-01-13T11:53:23.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.165+0000 7f1509d9b700 1 --2- 172.21.15.123:0/1085933503 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f14f402ff60 0x7f14f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:23.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.165+0000 7f1509d9b700 1 --2- 172.21.15.123:0/1085933503 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f14f402ff60 0x7f14f4032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f15000096a0 tx=0x7f1500006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:23.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.169+0000 7f14faffd700 1 -- 172.21.15.123:0/1085933503 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14fc014420 con 0x7f150c0601e0 2024-01-13T11:53:23.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.378+0000 7f150b59e700 1 -- 172.21.15.123:0/1085933503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f150c107f30 con 0x7f150c0601e0 2024-01-13T11:53:23.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.378+0000 7f14faffd700 1 -- 172.21.15.123:0/1085933503 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f14fc010230 con 0x7f150c0601e0 2024-01-13T11:53:23.367 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:23.367 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:23.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.380+0000 7f150b59e700 1 -- 172.21.15.123:0/1085933503 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f14f402ff60 msgr2=0x7f14f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:23.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.380+0000 7f150b59e700 1 --2- 172.21.15.123:0/1085933503 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f14f402ff60 0x7f14f4032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f15000096a0 tx=0x7f1500006b40).stop 2024-01-13T11:53:23.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.380+0000 7f150b59e700 1 -- 172.21.15.123:0/1085933503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f150c0601e0 msgr2=0x7f150c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:23.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.380+0000 7f150b59e700 1 --2- 172.21.15.123:0/1085933503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f150c0601e0 0x7f150c106500 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f14fc013510 tx=0x7f14fc00b040).stop 2024-01-13T11:53:23.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.380+0000 7f150b59e700 1 -- 172.21.15.123:0/1085933503 shutdown_connections 2024-01-13T11:53:23.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.380+0000 7f150b59e700 1 --2- 172.21.15.123:0/1085933503 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f14f402ff60 0x7f14f4032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:23.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.380+0000 7f150b59e700 1 --2- 172.21.15.123:0/1085933503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f150c0601e0 0x7f150c106500 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:23.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.380+0000 7f150b59e700 1 -- 172.21.15.123:0/1085933503 >> 172.21.15.123:0/1085933503 conn(0x7f150c0f09a0 msgr2=0x7f150c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:23.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.380+0000 7f150b59e700 1 -- 172.21.15.123:0/1085933503 shutdown_connections 2024-01-13T11:53:23.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:23.380+0000 7f150b59e700 1 -- 172.21.15.123:0/1085933503 wait complete. 2024-01-13T11:53:23.370 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:23.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:23 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:21.534161+0000 mgr.smithi059.hmyldj (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:24.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:24 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:23.379594+0000 mon.smithi059 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.123:0/1085933503' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:24.737 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:24.737 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:25.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:25 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:23.534544+0000 mgr.smithi059.hmyldj (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:25.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.876+0000 7fb3b3fff700 1 -- 172.21.15.123:0/4246513471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40f51a0 msgr2=0x7fb3b40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:25.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.876+0000 7fb3b3fff700 1 --2- 172.21.15.123:0/4246513471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40f51a0 0x7fb3b40f55c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fb3a4009fb0 tx=0x7fb3a4009b30).stop 2024-01-13T11:53:25.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.876+0000 7fb3b3fff700 1 -- 172.21.15.123:0/4246513471 shutdown_connections 2024-01-13T11:53:25.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.876+0000 7fb3b3fff700 1 --2- 172.21.15.123:0/4246513471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40f51a0 0x7fb3b40f55c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:25.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.876+0000 7fb3b3fff700 1 -- 172.21.15.123:0/4246513471 >> 172.21.15.123:0/4246513471 conn(0x7fb3b40f09a0 msgr2=0x7fb3b40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:25.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.876+0000 7fb3b3fff700 1 -- 172.21.15.123:0/4246513471 shutdown_connections 2024-01-13T11:53:25.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.876+0000 7fb3b3fff700 1 -- 172.21.15.123:0/4246513471 wait complete. 2024-01-13T11:53:25.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.876+0000 7fb3b3fff700 1 Processor -- start 2024-01-13T11:53:25.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.877+0000 7fb3b3fff700 1 -- start start 2024-01-13T11:53:25.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.877+0000 7fb3b3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40f51a0 0x7fb3b41053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:25.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.877+0000 7fb3b3fff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b41058a0 con 0x7fb3b40f51a0 2024-01-13T11:53:25.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.877+0000 7fb3b2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40f51a0 0x7fb3b41053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:25.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.877+0000 7fb3b2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40f51a0 0x7fb3b41053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:51386/0 (socket says 172.21.15.123:51386) 2024-01-13T11:53:25.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.877+0000 7fb3b2ffd700 1 -- 172.21.15.123:0/2943810828 learned_addr learned my addr 172.21.15.123:0/2943810828 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:25.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.877+0000 7fb3b2ffd700 1 -- 172.21.15.123:0/2943810828 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a4005040 con 0x7fb3b40f51a0 2024-01-13T11:53:25.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.877+0000 7fb3b2ffd700 1 --2- 172.21.15.123:0/2943810828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40f51a0 0x7fb3b41053b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fb3a4013510 tx=0x7fb3a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:25.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.878+0000 7fb39b7fe700 1 -- 172.21.15.123:0/2943810828 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3a4009320 con 0x7fb3b40f51a0 2024-01-13T11:53:25.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.878+0000 7fb39b7fe700 1 -- 172.21.15.123:0/2943810828 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3a4013f00 con 0x7fb3b40f51a0 2024-01-13T11:53:25.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.878+0000 7fb3b3fff700 1 -- 172.21.15.123:0/2943810828 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b4105aa0 con 0x7fb3b40f51a0 2024-01-13T11:53:25.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.878+0000 7fb39b7fe700 1 -- 172.21.15.123:0/2943810828 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3a4014dc0 con 0x7fb3b40f51a0 2024-01-13T11:53:25.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.878+0000 7fb3b3fff700 1 -- 172.21.15.123:0/2943810828 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b4105f30 con 0x7fb3b40f51a0 2024-01-13T11:53:25.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.878+0000 7fb39b7fe700 1 -- 172.21.15.123:0/2943810828 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3a401d3d0 con 0x7fb3b40f51a0 2024-01-13T11:53:25.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.879+0000 7fb39b7fe700 1 --2- 172.21.15.123:0/2943810828 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb39c02ff60 0x7fb39c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:25.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.879+0000 7fb39b7fe700 1 -- 172.21.15.123:0/2943810828 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3a40410f0 con 0x7fb3b40f51a0 2024-01-13T11:53:25.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.879+0000 7fb3b3fff700 1 -- 172.21.15.123:0/2943810828 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b4045ad0 con 0x7fb3b40f51a0 2024-01-13T11:53:25.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.879+0000 7fb3b27fc700 1 --2- 172.21.15.123:0/2943810828 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb39c02ff60 0x7fb39c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:25.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.879+0000 7fb3b27fc700 1 --2- 172.21.15.123:0/2943810828 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb39c02ff60 0x7fb39c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fb3a80096a0 tx=0x7fb3a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:25.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:25.884+0000 7fb39b7fe700 1 -- 172.21.15.123:0/2943810828 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3a4012920 con 0x7fb3b40f51a0 2024-01-13T11:53:26.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:26.092+0000 7fb3b3fff700 1 -- 172.21.15.123:0/2943810828 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3b4106de0 con 0x7fb3b40f51a0 2024-01-13T11:53:26.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:26.093+0000 7fb39b7fe700 1 -- 172.21.15.123:0/2943810828 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb3a402b020 con 0x7fb3b40f51a0 2024-01-13T11:53:26.082 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:26.083 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:26.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:26.095+0000 7fb3b3fff700 1 -- 172.21.15.123:0/2943810828 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb39c02ff60 msgr2=0x7fb39c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:26.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:26.095+0000 7fb3b3fff700 1 --2- 172.21.15.123:0/2943810828 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb39c02ff60 0x7fb39c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fb3a80096a0 tx=0x7fb3a8006b40).stop 2024-01-13T11:53:26.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:26.095+0000 7fb3b3fff700 1 -- 172.21.15.123:0/2943810828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40f51a0 msgr2=0x7fb3b41053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:26.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:26.095+0000 7fb3b3fff700 1 --2- 172.21.15.123:0/2943810828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40f51a0 0x7fb3b41053b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fb3a4013510 tx=0x7fb3a400b040).stop 2024-01-13T11:53:26.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:26.095+0000 7fb3b3fff700 1 -- 172.21.15.123:0/2943810828 shutdown_connections 2024-01-13T11:53:26.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:26.095+0000 7fb3b3fff700 1 --2- 172.21.15.123:0/2943810828 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb39c02ff60 0x7fb39c032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:26.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:26.095+0000 7fb3b3fff700 1 --2- 172.21.15.123:0/2943810828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40f51a0 0x7fb3b41053b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:26.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:26.095+0000 7fb3b3fff700 1 -- 172.21.15.123:0/2943810828 >> 172.21.15.123:0/2943810828 conn(0x7fb3b40f09a0 msgr2=0x7fb3b40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:26.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:26.095+0000 7fb3b3fff700 1 -- 172.21.15.123:0/2943810828 shutdown_connections 2024-01-13T11:53:26.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:26.095+0000 7fb3b3fff700 1 -- 172.21.15.123:0/2943810828 wait complete. 2024-01-13T11:53:26.086 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:26.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:26 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:26.094558+0000 mon.smithi059 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.123:0/2943810828' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:27.465 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:27.465 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:27.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:27 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:25.534815+0000 mgr.smithi059.hmyldj (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:28.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.612+0000 7fd986761700 1 -- 172.21.15.123:0/3865394521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9800601e0 msgr2=0x7fd9800f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:28.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.612+0000 7fd986761700 1 --2- 172.21.15.123:0/3865394521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9800601e0 0x7fd9800f9190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fd974009fb0 tx=0x7fd974009b30).stop 2024-01-13T11:53:28.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.612+0000 7fd986761700 1 -- 172.21.15.123:0/3865394521 shutdown_connections 2024-01-13T11:53:28.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.612+0000 7fd986761700 1 --2- 172.21.15.123:0/3865394521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9800601e0 0x7fd9800f9190 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:28.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.612+0000 7fd986761700 1 -- 172.21.15.123:0/3865394521 >> 172.21.15.123:0/3865394521 conn(0x7fd9800f0b70 msgr2=0x7fd9800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:28.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.612+0000 7fd986761700 1 -- 172.21.15.123:0/3865394521 shutdown_connections 2024-01-13T11:53:28.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.612+0000 7fd986761700 1 -- 172.21.15.123:0/3865394521 wait complete. 2024-01-13T11:53:28.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.613+0000 7fd986761700 1 Processor -- start 2024-01-13T11:53:28.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.613+0000 7fd986761700 1 -- start start 2024-01-13T11:53:28.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.613+0000 7fd986761700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9800601e0 0x7fd9801066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:28.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.613+0000 7fd986761700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd980106bd0 con 0x7fd9800601e0 2024-01-13T11:53:28.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.613+0000 7fd97ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9800601e0 0x7fd9801066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:28.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.613+0000 7fd97ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9800601e0 0x7fd9801066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:48106/0 (socket says 172.21.15.123:48106) 2024-01-13T11:53:28.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.613+0000 7fd97ffff700 1 -- 172.21.15.123:0/3719300950 learned_addr learned my addr 172.21.15.123:0/3719300950 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:28.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.614+0000 7fd97ffff700 1 -- 172.21.15.123:0/3719300950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd974005040 con 0x7fd9800601e0 2024-01-13T11:53:28.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.614+0000 7fd97ffff700 1 --2- 172.21.15.123:0/3719300950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9800601e0 0x7fd9801066e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fd974004dd0 tx=0x7fd974009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:28.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.614+0000 7fd97cff9700 1 -- 172.21.15.123:0/3719300950 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd974018070 con 0x7fd9800601e0 2024-01-13T11:53:28.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.614+0000 7fd986761700 1 -- 172.21.15.123:0/3719300950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd980106dd0 con 0x7fd9800601e0 2024-01-13T11:53:28.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.614+0000 7fd97cff9700 1 -- 172.21.15.123:0/3719300950 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd974013410 con 0x7fd9800601e0 2024-01-13T11:53:28.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.614+0000 7fd97cff9700 1 -- 172.21.15.123:0/3719300950 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd97401c3c0 con 0x7fd9800601e0 2024-01-13T11:53:28.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.614+0000 7fd986761700 1 -- 172.21.15.123:0/3719300950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd980107260 con 0x7fd9800601e0 2024-01-13T11:53:28.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.615+0000 7fd986761700 1 -- 172.21.15.123:0/3719300950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd960004fa0 con 0x7fd9800601e0 2024-01-13T11:53:28.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.615+0000 7fd97cff9700 1 -- 172.21.15.123:0/3719300950 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd97401c520 con 0x7fd9800601e0 2024-01-13T11:53:28.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.615+0000 7fd97cff9700 1 --2- 172.21.15.123:0/3719300950 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd96802ffb0 0x7fd968032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:28.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.615+0000 7fd97cff9700 1 -- 172.21.15.123:0/3719300950 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd974041260 con 0x7fd9800601e0 2024-01-13T11:53:28.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.616+0000 7fd97f7fe700 1 --2- 172.21.15.123:0/3719300950 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd96802ffb0 0x7fd968032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:28.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.616+0000 7fd97f7fe700 1 --2- 172.21.15.123:0/3719300950 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd96802ffb0 0x7fd968032470 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fd9700096a0 tx=0x7fd970006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:28.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.620+0000 7fd97cff9700 1 -- 172.21.15.123:0/3719300950 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd97401c8e0 con 0x7fd9800601e0 2024-01-13T11:53:28.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.826+0000 7fd986761700 1 -- 172.21.15.123:0/3719300950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd960005670 con 0x7fd9800601e0 2024-01-13T11:53:28.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.827+0000 7fd97cff9700 1 -- 172.21.15.123:0/3719300950 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd974004490 con 0x7fd9800601e0 2024-01-13T11:53:28.816 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:28.816 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:28.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.828+0000 7fd986761700 1 -- 172.21.15.123:0/3719300950 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd96802ffb0 msgr2=0x7fd968032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:28.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.828+0000 7fd986761700 1 --2- 172.21.15.123:0/3719300950 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd96802ffb0 0x7fd968032470 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fd9700096a0 tx=0x7fd970006b40).stop 2024-01-13T11:53:28.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.828+0000 7fd986761700 1 -- 172.21.15.123:0/3719300950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9800601e0 msgr2=0x7fd9801066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:28.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.828+0000 7fd986761700 1 --2- 172.21.15.123:0/3719300950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9800601e0 0x7fd9801066e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fd974004dd0 tx=0x7fd974009b30).stop 2024-01-13T11:53:28.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.829+0000 7fd986761700 1 -- 172.21.15.123:0/3719300950 shutdown_connections 2024-01-13T11:53:28.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.829+0000 7fd986761700 1 --2- 172.21.15.123:0/3719300950 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd96802ffb0 0x7fd968032470 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:28.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.829+0000 7fd986761700 1 --2- 172.21.15.123:0/3719300950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9800601e0 0x7fd9801066e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:28.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.829+0000 7fd986761700 1 -- 172.21.15.123:0/3719300950 >> 172.21.15.123:0/3719300950 conn(0x7fd9800f0b70 msgr2=0x7fd9800f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:28.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.829+0000 7fd986761700 1 -- 172.21.15.123:0/3719300950 shutdown_connections 2024-01-13T11:53:28.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:28.829+0000 7fd986761700 1 -- 172.21.15.123:0/3719300950 wait complete. 2024-01-13T11:53:28.818 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:29.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:29 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:27.535138+0000 mgr.smithi059.hmyldj (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:29.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:29 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:28.828289+0000 mon.smithi059 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.123:0/3719300950' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:30.230 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:30.231 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:31.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.386+0000 7fbfd1d9d700 1 -- 172.21.15.123:0/657412660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfcc0f6510 msgr2=0x7fbfcc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:31.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.386+0000 7fbfd1d9d700 1 --2- 172.21.15.123:0/657412660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfcc0f6510 0x7fbfcc0f6930 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fbfc0009fb0 tx=0x7fbfc0009b30).stop 2024-01-13T11:53:31.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.386+0000 7fbfd1d9d700 1 -- 172.21.15.123:0/657412660 shutdown_connections 2024-01-13T11:53:31.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.386+0000 7fbfd1d9d700 1 --2- 172.21.15.123:0/657412660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfcc0f6510 0x7fbfcc0f6930 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:31.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.386+0000 7fbfd1d9d700 1 -- 172.21.15.123:0/657412660 >> 172.21.15.123:0/657412660 conn(0x7fbfcc0f1c70 msgr2=0x7fbfcc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:31.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.386+0000 7fbfd1d9d700 1 -- 172.21.15.123:0/657412660 shutdown_connections 2024-01-13T11:53:31.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.386+0000 7fbfd1d9d700 1 -- 172.21.15.123:0/657412660 wait complete. 2024-01-13T11:53:31.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.387+0000 7fbfd1d9d700 1 Processor -- start 2024-01-13T11:53:31.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.387+0000 7fbfd1d9d700 1 -- start start 2024-01-13T11:53:31.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.387+0000 7fbfd1d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfcc0f6510 0x7fbfcc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:31.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.387+0000 7fbfd1d9d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfcc106c20 con 0x7fbfcc0f6510 2024-01-13T11:53:31.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.387+0000 7fbfcb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfcc0f6510 0x7fbfcc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:31.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.387+0000 7fbfcb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfcc0f6510 0x7fbfcc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:48130/0 (socket says 172.21.15.123:48130) 2024-01-13T11:53:31.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.387+0000 7fbfcb7fe700 1 -- 172.21.15.123:0/535248192 learned_addr learned my addr 172.21.15.123:0/535248192 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:31.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.387+0000 7fbfcb7fe700 1 -- 172.21.15.123:0/535248192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc0005040 con 0x7fbfcc0f6510 2024-01-13T11:53:31.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.388+0000 7fbfcb7fe700 1 --2- 172.21.15.123:0/535248192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfcc0f6510 0x7fbfcc106730 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fbfc0013510 tx=0x7fbfc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:31.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.388+0000 7fbfb3fff700 1 -- 172.21.15.123:0/535248192 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfc0009320 con 0x7fbfcc0f6510 2024-01-13T11:53:31.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.388+0000 7fbfd1d9d700 1 -- 172.21.15.123:0/535248192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfcc106e20 con 0x7fbfcc0f6510 2024-01-13T11:53:31.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.388+0000 7fbfb3fff700 1 -- 172.21.15.123:0/535248192 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbfc0013f00 con 0x7fbfcc0f6510 2024-01-13T11:53:31.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.388+0000 7fbfb3fff700 1 -- 172.21.15.123:0/535248192 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfc0014d90 con 0x7fbfcc0f6510 2024-01-13T11:53:31.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.388+0000 7fbfd1d9d700 1 -- 172.21.15.123:0/535248192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfcc1072b0 con 0x7fbfcc0f6510 2024-01-13T11:53:31.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.389+0000 7fbfb3fff700 1 -- 172.21.15.123:0/535248192 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbfc00095f0 con 0x7fbfcc0f6510 2024-01-13T11:53:31.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.389+0000 7fbfb3fff700 1 --2- 172.21.15.123:0/535248192 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbfb402ff60 0x7fbfb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:31.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.389+0000 7fbfd1d9d700 1 -- 172.21.15.123:0/535248192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfcc045ad0 con 0x7fbfcc0f6510 2024-01-13T11:53:31.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.389+0000 7fbfb3fff700 1 -- 172.21.15.123:0/535248192 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbfc0040fe0 con 0x7fbfcc0f6510 2024-01-13T11:53:31.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.389+0000 7fbfcaffd700 1 --2- 172.21.15.123:0/535248192 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbfb402ff60 0x7fbfb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:31.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.389+0000 7fbfcaffd700 1 --2- 172.21.15.123:0/535248192 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbfb402ff60 0x7fbfb4032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fbfbc0096a0 tx=0x7fbfbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:31.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.393+0000 7fbfb3fff700 1 -- 172.21.15.123:0/535248192 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbfc001d7f0 con 0x7fbfcc0f6510 2024-01-13T11:53:31.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.600+0000 7fbfd1d9d700 1 -- 172.21.15.123:0/535248192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbfcc0fc050 con 0x7fbfcc0f6510 2024-01-13T11:53:31.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.601+0000 7fbfb3fff700 1 -- 172.21.15.123:0/535248192 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbfc0010230 con 0x7fbfcc0f6510 2024-01-13T11:53:31.590 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:31.590 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:31.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.602+0000 7fbfd1d9d700 1 -- 172.21.15.123:0/535248192 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbfb402ff60 msgr2=0x7fbfb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:31.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.602+0000 7fbfd1d9d700 1 --2- 172.21.15.123:0/535248192 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbfb402ff60 0x7fbfb4032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fbfbc0096a0 tx=0x7fbfbc006b40).stop 2024-01-13T11:53:31.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.602+0000 7fbfd1d9d700 1 -- 172.21.15.123:0/535248192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfcc0f6510 msgr2=0x7fbfcc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:31.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.602+0000 7fbfd1d9d700 1 --2- 172.21.15.123:0/535248192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfcc0f6510 0x7fbfcc106730 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fbfc0013510 tx=0x7fbfc000b040).stop 2024-01-13T11:53:31.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.602+0000 7fbfd1d9d700 1 -- 172.21.15.123:0/535248192 shutdown_connections 2024-01-13T11:53:31.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.602+0000 7fbfd1d9d700 1 --2- 172.21.15.123:0/535248192 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbfb402ff60 0x7fbfb4032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:31.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.602+0000 7fbfd1d9d700 1 --2- 172.21.15.123:0/535248192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfcc0f6510 0x7fbfcc106730 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:31.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.602+0000 7fbfd1d9d700 1 -- 172.21.15.123:0/535248192 >> 172.21.15.123:0/535248192 conn(0x7fbfcc0f1c70 msgr2=0x7fbfcc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:31.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.602+0000 7fbfd1d9d700 1 -- 172.21.15.123:0/535248192 shutdown_connections 2024-01-13T11:53:31.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:31.602+0000 7fbfd1d9d700 1 -- 172.21.15.123:0/535248192 wait complete. 2024-01-13T11:53:31.592 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:31.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:31 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:29.535506+0000 mgr.smithi059.hmyldj (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:32.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:32 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:31.602003+0000 mon.smithi059 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.123:0/535248192' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:32.976 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:32.976 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:33.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:33 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:31.535877+0000 mgr.smithi059.hmyldj (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:34.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.149+0000 7f869565d700 1 -- 172.21.15.123:0/1234596764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86900f6480 msgr2=0x7f86900f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:34.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.149+0000 7f869565d700 1 --2- 172.21.15.123:0/1234596764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86900f6480 0x7f86900f68a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f8684009fb0 tx=0x7f8684009b30).stop 2024-01-13T11:53:34.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.149+0000 7f869565d700 1 -- 172.21.15.123:0/1234596764 shutdown_connections 2024-01-13T11:53:34.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.149+0000 7f869565d700 1 --2- 172.21.15.123:0/1234596764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86900f6480 0x7f86900f68a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:34.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.149+0000 7f869565d700 1 -- 172.21.15.123:0/1234596764 >> 172.21.15.123:0/1234596764 conn(0x7f86900f1c00 msgr2=0x7f86900f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:34.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.149+0000 7f869565d700 1 -- 172.21.15.123:0/1234596764 shutdown_connections 2024-01-13T11:53:34.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.149+0000 7f869565d700 1 -- 172.21.15.123:0/1234596764 wait complete. 2024-01-13T11:53:34.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.149+0000 7f869565d700 1 Processor -- start 2024-01-13T11:53:34.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.150+0000 7f869565d700 1 -- start start 2024-01-13T11:53:34.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.150+0000 7f869565d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86900f6480 0x7f86901066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:34.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.150+0000 7f869565d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8690106bc0 con 0x7f86900f6480 2024-01-13T11:53:34.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.150+0000 7f868effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86900f6480 0x7f86901066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:34.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.150+0000 7f868effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86900f6480 0x7f86901066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:48150/0 (socket says 172.21.15.123:48150) 2024-01-13T11:53:34.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.150+0000 7f868effd700 1 -- 172.21.15.123:0/283370692 learned_addr learned my addr 172.21.15.123:0/283370692 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:34.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.150+0000 7f868effd700 1 -- 172.21.15.123:0/283370692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8684005040 con 0x7f86900f6480 2024-01-13T11:53:34.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.151+0000 7f868effd700 1 --2- 172.21.15.123:0/283370692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86900f6480 0x7f86901066d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f8684014000 tx=0x7f868400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:34.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.151+0000 7f86777fe700 1 -- 172.21.15.123:0/283370692 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86840095f0 con 0x7f86900f6480 2024-01-13T11:53:34.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.151+0000 7f86777fe700 1 -- 172.21.15.123:0/283370692 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8684007e70 con 0x7f86900f6480 2024-01-13T11:53:34.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.151+0000 7f869565d700 1 -- 172.21.15.123:0/283370692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8690106dc0 con 0x7f86900f6480 2024-01-13T11:53:34.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.151+0000 7f86777fe700 1 -- 172.21.15.123:0/283370692 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8684014ea0 con 0x7f86900f6480 2024-01-13T11:53:34.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.151+0000 7f869565d700 1 -- 172.21.15.123:0/283370692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8690107250 con 0x7f86900f6480 2024-01-13T11:53:34.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.152+0000 7f86777fe700 1 -- 172.21.15.123:0/283370692 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8684025070 con 0x7f86900f6480 2024-01-13T11:53:34.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.152+0000 7f869565d700 1 -- 172.21.15.123:0/283370692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8690045ad0 con 0x7f86900f6480 2024-01-13T11:53:34.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.152+0000 7f86777fe700 1 --2- 172.21.15.123:0/283370692 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f867802ff60 0x7f8678032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:34.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.152+0000 7f86777fe700 1 -- 172.21.15.123:0/283370692 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f86840409b0 con 0x7f86900f6480 2024-01-13T11:53:34.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.152+0000 7f868e7fc700 1 --2- 172.21.15.123:0/283370692 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f867802ff60 0x7f8678032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:34.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.152+0000 7f868e7fc700 1 --2- 172.21.15.123:0/283370692 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f867802ff60 0x7f8678032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f86800096a0 tx=0x7f8680006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:34.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.156+0000 7f86777fe700 1 -- 172.21.15.123:0/283370692 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f86840079c0 con 0x7f86900f6480 2024-01-13T11:53:34.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.361+0000 7f869565d700 1 -- 172.21.15.123:0/283370692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8690108100 con 0x7f86900f6480 2024-01-13T11:53:34.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.361+0000 7f86777fe700 1 -- 172.21.15.123:0/283370692 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f868400f300 con 0x7f86900f6480 2024-01-13T11:53:34.350 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:34.350 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:34.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.363+0000 7f869565d700 1 -- 172.21.15.123:0/283370692 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f867802ff60 msgr2=0x7f8678032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:34.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.363+0000 7f869565d700 1 --2- 172.21.15.123:0/283370692 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f867802ff60 0x7f8678032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f86800096a0 tx=0x7f8680006b40).stop 2024-01-13T11:53:34.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.363+0000 7f869565d700 1 -- 172.21.15.123:0/283370692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86900f6480 msgr2=0x7f86901066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:34.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.363+0000 7f869565d700 1 --2- 172.21.15.123:0/283370692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86900f6480 0x7f86901066d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f8684014000 tx=0x7f868400b040).stop 2024-01-13T11:53:34.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.363+0000 7f869565d700 1 -- 172.21.15.123:0/283370692 shutdown_connections 2024-01-13T11:53:34.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.363+0000 7f869565d700 1 --2- 172.21.15.123:0/283370692 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f867802ff60 0x7f8678032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:34.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.363+0000 7f869565d700 1 --2- 172.21.15.123:0/283370692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86900f6480 0x7f86901066d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:34.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.363+0000 7f869565d700 1 -- 172.21.15.123:0/283370692 >> 172.21.15.123:0/283370692 conn(0x7f86900f1c00 msgr2=0x7f86900f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:34.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.363+0000 7f869565d700 1 -- 172.21.15.123:0/283370692 shutdown_connections 2024-01-13T11:53:34.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:34.363+0000 7f869565d700 1 -- 172.21.15.123:0/283370692 wait complete. 2024-01-13T11:53:34.352 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:35.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:35 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:33.536209+0000 mgr.smithi059.hmyldj (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:35.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:35 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:34.362478+0000 mon.smithi059 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.123:0/283370692' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:35.738 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:35.738 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:36.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.886+0000 7f7b174af700 1 -- 172.21.15.123:0/3814626211 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b100f6510 msgr2=0x7f7b100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:36.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.886+0000 7f7b174af700 1 --2- 172.21.15.123:0/3814626211 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b100f6510 0x7f7b100f6930 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f7b00009fb0 tx=0x7f7b00009b30).stop 2024-01-13T11:53:36.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.886+0000 7f7b174af700 1 -- 172.21.15.123:0/3814626211 shutdown_connections 2024-01-13T11:53:36.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.886+0000 7f7b174af700 1 --2- 172.21.15.123:0/3814626211 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b100f6510 0x7f7b100f6930 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:36.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.886+0000 7f7b174af700 1 -- 172.21.15.123:0/3814626211 >> 172.21.15.123:0/3814626211 conn(0x7f7b100f1c70 msgr2=0x7f7b100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:36.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.886+0000 7f7b174af700 1 -- 172.21.15.123:0/3814626211 shutdown_connections 2024-01-13T11:53:36.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.886+0000 7f7b174af700 1 -- 172.21.15.123:0/3814626211 wait complete. 2024-01-13T11:53:36.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.886+0000 7f7b174af700 1 Processor -- start 2024-01-13T11:53:36.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.887+0000 7f7b174af700 1 -- start start 2024-01-13T11:53:36.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.887+0000 7f7b174af700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b100f6510 0x7f7b10106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:36.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.887+0000 7f7b174af700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b10106c20 con 0x7f7b100f6510 2024-01-13T11:53:36.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.887+0000 7f7b1524b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b100f6510 0x7f7b10106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:36.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.887+0000 7f7b1524b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b100f6510 0x7f7b10106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:49608/0 (socket says 172.21.15.123:49608) 2024-01-13T11:53:36.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.887+0000 7f7b1524b700 1 -- 172.21.15.123:0/4186428838 learned_addr learned my addr 172.21.15.123:0/4186428838 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:36.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.887+0000 7f7b1524b700 1 -- 172.21.15.123:0/4186428838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b00005040 con 0x7f7b100f6510 2024-01-13T11:53:36.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.887+0000 7f7b1524b700 1 --2- 172.21.15.123:0/4186428838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b100f6510 0x7f7b10106730 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f7b00013510 tx=0x7f7b0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:36.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.887+0000 7f7b05ffb700 1 -- 172.21.15.123:0/4186428838 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b00009320 con 0x7f7b100f6510 2024-01-13T11:53:36.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.888+0000 7f7b05ffb700 1 -- 172.21.15.123:0/4186428838 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b00013f00 con 0x7f7b100f6510 2024-01-13T11:53:36.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.888+0000 7f7b174af700 1 -- 172.21.15.123:0/4186428838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b10106e20 con 0x7f7b100f6510 2024-01-13T11:53:36.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.888+0000 7f7b05ffb700 1 -- 172.21.15.123:0/4186428838 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b00014d90 con 0x7f7b100f6510 2024-01-13T11:53:36.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.888+0000 7f7b174af700 1 -- 172.21.15.123:0/4186428838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b101072b0 con 0x7f7b100f6510 2024-01-13T11:53:36.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.888+0000 7f7b05ffb700 1 -- 172.21.15.123:0/4186428838 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b000095f0 con 0x7f7b100f6510 2024-01-13T11:53:36.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.889+0000 7f7b05ffb700 1 --2- 172.21.15.123:0/4186428838 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7afc02ff60 0x7f7afc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:36.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.889+0000 7f7b05ffb700 1 -- 172.21.15.123:0/4186428838 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b00041000 con 0x7f7b100f6510 2024-01-13T11:53:36.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.889+0000 7f7b174af700 1 -- 172.21.15.123:0/4186428838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b10059100 con 0x7f7b100f6510 2024-01-13T11:53:36.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.889+0000 7f7b14a4a700 1 --2- 172.21.15.123:0/4186428838 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7afc02ff60 0x7f7afc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:36.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.889+0000 7f7b14a4a700 1 --2- 172.21.15.123:0/4186428838 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7afc02ff60 0x7f7afc032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f7b0c0096a0 tx=0x7f7b0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:36.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:36.894+0000 7f7b05ffb700 1 -- 172.21.15.123:0/4186428838 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b0001d7f0 con 0x7f7b100f6510 2024-01-13T11:53:37.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:37.100+0000 7f7b174af700 1 -- 172.21.15.123:0/4186428838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b100fbf60 con 0x7f7b100f6510 2024-01-13T11:53:37.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:37.101+0000 7f7b05ffb700 1 -- 172.21.15.123:0/4186428838 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7b0002b020 con 0x7f7b100f6510 2024-01-13T11:53:37.089 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:37.089 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:37.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:37.102+0000 7f7b174af700 1 -- 172.21.15.123:0/4186428838 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7afc02ff60 msgr2=0x7f7afc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:37.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:37.102+0000 7f7b174af700 1 --2- 172.21.15.123:0/4186428838 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7afc02ff60 0x7f7afc032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f7b0c0096a0 tx=0x7f7b0c006b40).stop 2024-01-13T11:53:37.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:37.102+0000 7f7b174af700 1 -- 172.21.15.123:0/4186428838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b100f6510 msgr2=0x7f7b10106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:37.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:37.102+0000 7f7b174af700 1 --2- 172.21.15.123:0/4186428838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b100f6510 0x7f7b10106730 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f7b00013510 tx=0x7f7b0000b040).stop 2024-01-13T11:53:37.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:37.102+0000 7f7b174af700 1 -- 172.21.15.123:0/4186428838 shutdown_connections 2024-01-13T11:53:37.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:37.102+0000 7f7b174af700 1 --2- 172.21.15.123:0/4186428838 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7afc02ff60 0x7f7afc032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:37.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:37.102+0000 7f7b174af700 1 --2- 172.21.15.123:0/4186428838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b100f6510 0x7f7b10106730 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:37.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:37.102+0000 7f7b174af700 1 -- 172.21.15.123:0/4186428838 >> 172.21.15.123:0/4186428838 conn(0x7f7b100f1c70 msgr2=0x7f7b100f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:37.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:37.102+0000 7f7b174af700 1 -- 172.21.15.123:0/4186428838 shutdown_connections 2024-01-13T11:53:37.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:37.102+0000 7f7b174af700 1 -- 172.21.15.123:0/4186428838 wait complete. 2024-01-13T11:53:37.092 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:37.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:37 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:35.536625+0000 mgr.smithi059.hmyldj (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:37.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:37 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:37.101899+0000 mon.smithi059 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.123:0/4186428838' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:38.485 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:38.486 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:39.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.632+0000 7f08099d3700 1 -- 172.21.15.123:0/1906933200 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08040f5410 msgr2=0x7f08040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:39.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.632+0000 7f08099d3700 1 --2- 172.21.15.123:0/1906933200 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08040f5410 0x7f08040f5830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f07ec009fb0 tx=0x7f07ec009b30).stop 2024-01-13T11:53:39.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.632+0000 7f08099d3700 1 -- 172.21.15.123:0/1906933200 shutdown_connections 2024-01-13T11:53:39.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.632+0000 7f08099d3700 1 --2- 172.21.15.123:0/1906933200 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08040f5410 0x7f08040f5830 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:39.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.632+0000 7f08099d3700 1 -- 172.21.15.123:0/1906933200 >> 172.21.15.123:0/1906933200 conn(0x7f08040f0b70 msgr2=0x7f08040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:39.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.632+0000 7f08099d3700 1 -- 172.21.15.123:0/1906933200 shutdown_connections 2024-01-13T11:53:39.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.632+0000 7f08099d3700 1 -- 172.21.15.123:0/1906933200 wait complete. 2024-01-13T11:53:39.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.633+0000 7f08099d3700 1 Processor -- start 2024-01-13T11:53:39.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.633+0000 7f08099d3700 1 -- start start 2024-01-13T11:53:39.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.633+0000 7f08099d3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08040f5410 0x7f08040fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:39.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.633+0000 7f08099d3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08040fb820 con 0x7f08040f5410 2024-01-13T11:53:39.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.633+0000 7f0802ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08040f5410 0x7f08040fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:39.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.633+0000 7f0802ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08040f5410 0x7f08040fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:49632/0 (socket says 172.21.15.123:49632) 2024-01-13T11:53:39.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.633+0000 7f0802ffd700 1 -- 172.21.15.123:0/1778144594 learned_addr learned my addr 172.21.15.123:0/1778144594 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:39.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.633+0000 7f0802ffd700 1 -- 172.21.15.123:0/1778144594 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ec005040 con 0x7f08040f5410 2024-01-13T11:53:39.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.633+0000 7f0802ffd700 1 --2- 172.21.15.123:0/1778144594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08040f5410 0x7f08040fd3a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f07ec014000 tx=0x7f07ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:39.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.634+0000 7f07fb7fe700 1 -- 172.21.15.123:0/1778144594 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07ec0095f0 con 0x7f08040f5410 2024-01-13T11:53:39.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.634+0000 7f08099d3700 1 -- 172.21.15.123:0/1778144594 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08040fb9c0 con 0x7f08040f5410 2024-01-13T11:53:39.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.634+0000 7f07fb7fe700 1 -- 172.21.15.123:0/1778144594 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07ec007e70 con 0x7f08040f5410 2024-01-13T11:53:39.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.634+0000 7f07fb7fe700 1 -- 172.21.15.123:0/1778144594 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07ec014ea0 con 0x7f08040f5410 2024-01-13T11:53:39.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.634+0000 7f08099d3700 1 -- 172.21.15.123:0/1778144594 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08040fbe50 con 0x7f08040f5410 2024-01-13T11:53:39.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.634+0000 7f07fb7fe700 1 -- 172.21.15.123:0/1778144594 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07ec025070 con 0x7f08040f5410 2024-01-13T11:53:39.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.634+0000 7f08099d3700 1 -- 172.21.15.123:0/1778144594 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0804059100 con 0x7f08040f5410 2024-01-13T11:53:39.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.635+0000 7f07fb7fe700 1 --2- 172.21.15.123:0/1778144594 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f07f002ff60 0x7f07f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:39.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.635+0000 7f07fb7fe700 1 -- 172.21.15.123:0/1778144594 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07ec040970 con 0x7f08040f5410 2024-01-13T11:53:39.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.635+0000 7f08027fc700 1 --2- 172.21.15.123:0/1778144594 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f07f002ff60 0x7f07f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:39.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.635+0000 7f08027fc700 1 --2- 172.21.15.123:0/1778144594 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f07f002ff60 0x7f07f0032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f07f40096a0 tx=0x7f07f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:39.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.639+0000 7f07fb7fe700 1 -- 172.21.15.123:0/1778144594 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07ec0079c0 con 0x7f08040f5410 2024-01-13T11:53:39.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:39 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:37.536984+0000 mgr.smithi059.hmyldj (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:39.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.845+0000 7f08099d3700 1 -- 172.21.15.123:0/1778144594 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08040fae60 con 0x7f08040f5410 2024-01-13T11:53:39.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.846+0000 7f07fb7fe700 1 -- 172.21.15.123:0/1778144594 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f07ec00f300 con 0x7f08040f5410 2024-01-13T11:53:39.835 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:39.835 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:39.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.847+0000 7f08099d3700 1 -- 172.21.15.123:0/1778144594 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f07f002ff60 msgr2=0x7f07f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:39.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.847+0000 7f08099d3700 1 --2- 172.21.15.123:0/1778144594 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f07f002ff60 0x7f07f0032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f07f40096a0 tx=0x7f07f4006b40).stop 2024-01-13T11:53:39.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.847+0000 7f08099d3700 1 -- 172.21.15.123:0/1778144594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08040f5410 msgr2=0x7f08040fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:39.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.847+0000 7f08099d3700 1 --2- 172.21.15.123:0/1778144594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08040f5410 0x7f08040fd3a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f07ec014000 tx=0x7f07ec00b040).stop 2024-01-13T11:53:39.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.848+0000 7f08099d3700 1 -- 172.21.15.123:0/1778144594 shutdown_connections 2024-01-13T11:53:39.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.848+0000 7f08099d3700 1 --2- 172.21.15.123:0/1778144594 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f07f002ff60 0x7f07f0032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:39.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.848+0000 7f08099d3700 1 --2- 172.21.15.123:0/1778144594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08040f5410 0x7f08040fd3a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:39.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.848+0000 7f08099d3700 1 -- 172.21.15.123:0/1778144594 >> 172.21.15.123:0/1778144594 conn(0x7f08040f0b70 msgr2=0x7f080405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:39.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.848+0000 7f08099d3700 1 -- 172.21.15.123:0/1778144594 shutdown_connections 2024-01-13T11:53:39.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:39.848+0000 7f08099d3700 1 -- 172.21.15.123:0/1778144594 wait complete. 2024-01-13T11:53:39.837 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:40.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:39.640669+0000 mon.smithi059 (mon.0) 196 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/mirror_snapshot_schedule"}]: dispatch 2024-01-13T11:53:40.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:39.642055+0000 mon.smithi059 (mon.0) 197 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/trash_purge_schedule"}]: dispatch 2024-01-13T11:53:40.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:39.847067+0000 mon.smithi059 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.123:0/1778144594' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:41.221 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:41.221 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:41.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:41 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:39.537238+0000 mgr.smithi059.hmyldj (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:42.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.390+0000 7f943cc3a700 1 -- 172.21.15.123:0/1217242104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94380f6510 msgr2=0x7f94380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:42.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.390+0000 7f943cc3a700 1 --2- 172.21.15.123:0/1217242104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94380f6510 0x7f94380f6930 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f942c009fb0 tx=0x7f942c009b30).stop 2024-01-13T11:53:42.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.390+0000 7f943cc3a700 1 -- 172.21.15.123:0/1217242104 shutdown_connections 2024-01-13T11:53:42.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.390+0000 7f943cc3a700 1 --2- 172.21.15.123:0/1217242104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94380f6510 0x7f94380f6930 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:42.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.390+0000 7f943cc3a700 1 -- 172.21.15.123:0/1217242104 >> 172.21.15.123:0/1217242104 conn(0x7f94380f1c70 msgr2=0x7f94380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:42.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.390+0000 7f943cc3a700 1 -- 172.21.15.123:0/1217242104 shutdown_connections 2024-01-13T11:53:42.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.390+0000 7f943cc3a700 1 -- 172.21.15.123:0/1217242104 wait complete. 2024-01-13T11:53:42.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.391+0000 7f943cc3a700 1 Processor -- start 2024-01-13T11:53:42.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.391+0000 7f943cc3a700 1 -- start start 2024-01-13T11:53:42.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.391+0000 7f943cc3a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94380f6510 0x7f943806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:42.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.391+0000 7f943cc3a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9438069e20 con 0x7f94380f6510 2024-01-13T11:53:42.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.391+0000 7f943659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94380f6510 0x7f943806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:42.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.391+0000 7f943659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94380f6510 0x7f943806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:49658/0 (socket says 172.21.15.123:49658) 2024-01-13T11:53:42.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.391+0000 7f943659c700 1 -- 172.21.15.123:0/3968979960 learned_addr learned my addr 172.21.15.123:0/3968979960 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:42.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.392+0000 7f943659c700 1 -- 172.21.15.123:0/3968979960 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f942c005040 con 0x7f94380f6510 2024-01-13T11:53:42.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.392+0000 7f943659c700 1 --2- 172.21.15.123:0/3968979960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94380f6510 0x7f943806b9a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f942c013510 tx=0x7f942c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:42.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.392+0000 7f9426ffd700 1 -- 172.21.15.123:0/3968979960 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f942c009320 con 0x7f94380f6510 2024-01-13T11:53:42.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.392+0000 7f943cc3a700 1 -- 172.21.15.123:0/3968979960 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9438069fc0 con 0x7f94380f6510 2024-01-13T11:53:42.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.392+0000 7f9426ffd700 1 -- 172.21.15.123:0/3968979960 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f942c013f00 con 0x7f94380f6510 2024-01-13T11:53:42.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.392+0000 7f9426ffd700 1 -- 172.21.15.123:0/3968979960 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f942c014d90 con 0x7f94380f6510 2024-01-13T11:53:42.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.392+0000 7f943cc3a700 1 -- 172.21.15.123:0/3968979960 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f943806a450 con 0x7f94380f6510 2024-01-13T11:53:42.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.393+0000 7f9426ffd700 1 -- 172.21.15.123:0/3968979960 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f942c0095f0 con 0x7f94380f6510 2024-01-13T11:53:42.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.393+0000 7f943cc3a700 1 -- 172.21.15.123:0/3968979960 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9438059100 con 0x7f94380f6510 2024-01-13T11:53:42.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.393+0000 7f9426ffd700 1 --2- 172.21.15.123:0/3968979960 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f942002ff60 0x7f9420032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:42.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.393+0000 7f9426ffd700 1 -- 172.21.15.123:0/3968979960 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f942c041080 con 0x7f94380f6510 2024-01-13T11:53:42.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.393+0000 7f9435d9b700 1 --2- 172.21.15.123:0/3968979960 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f942002ff60 0x7f9420032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:42.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.393+0000 7f9435d9b700 1 --2- 172.21.15.123:0/3968979960 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f942002ff60 0x7f9420032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f94280096a0 tx=0x7f9428006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:42.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.398+0000 7f9426ffd700 1 -- 172.21.15.123:0/3968979960 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f942c014420 con 0x7f94380f6510 2024-01-13T11:53:42.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.604+0000 7f943cc3a700 1 -- 172.21.15.123:0/3968979960 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94380fbf60 con 0x7f94380f6510 2024-01-13T11:53:42.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.605+0000 7f9426ffd700 1 -- 172.21.15.123:0/3968979960 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f942c010230 con 0x7f94380f6510 2024-01-13T11:53:42.594 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:42.594 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:42.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.606+0000 7f943cc3a700 1 -- 172.21.15.123:0/3968979960 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f942002ff60 msgr2=0x7f9420032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:42.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.606+0000 7f943cc3a700 1 --2- 172.21.15.123:0/3968979960 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f942002ff60 0x7f9420032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f94280096a0 tx=0x7f9428006b40).stop 2024-01-13T11:53:42.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.606+0000 7f943cc3a700 1 -- 172.21.15.123:0/3968979960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94380f6510 msgr2=0x7f943806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:42.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.606+0000 7f943cc3a700 1 --2- 172.21.15.123:0/3968979960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94380f6510 0x7f943806b9a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f942c013510 tx=0x7f942c00b040).stop 2024-01-13T11:53:42.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.607+0000 7f943cc3a700 1 -- 172.21.15.123:0/3968979960 shutdown_connections 2024-01-13T11:53:42.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.607+0000 7f943cc3a700 1 --2- 172.21.15.123:0/3968979960 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f942002ff60 0x7f9420032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:42.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.607+0000 7f943cc3a700 1 --2- 172.21.15.123:0/3968979960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94380f6510 0x7f943806b9a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:42.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.607+0000 7f943cc3a700 1 -- 172.21.15.123:0/3968979960 >> 172.21.15.123:0/3968979960 conn(0x7f94380f1c70 msgr2=0x7f94380f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:42.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.607+0000 7f943cc3a700 1 -- 172.21.15.123:0/3968979960 shutdown_connections 2024-01-13T11:53:42.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:42.607+0000 7f943cc3a700 1 -- 172.21.15.123:0/3968979960 wait complete. 2024-01-13T11:53:42.596 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:43.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:43 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:41.537580+0000 mgr.smithi059.hmyldj (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:43.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:43 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:42.606334+0000 mon.smithi059 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.123:0/3968979960' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:43.972 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:43.972 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:45.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.163+0000 7f76950ac700 1 -- 172.21.15.123:0/2377041048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76900601e0 msgr2=0x7f76900f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:45.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.163+0000 7f76950ac700 1 --2- 172.21.15.123:0/2377041048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76900601e0 0x7f76900f9190 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f7678009fb0 tx=0x7f7678009b30).stop 2024-01-13T11:53:45.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.163+0000 7f76950ac700 1 -- 172.21.15.123:0/2377041048 shutdown_connections 2024-01-13T11:53:45.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.163+0000 7f76950ac700 1 --2- 172.21.15.123:0/2377041048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76900601e0 0x7f76900f9190 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:45.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.163+0000 7f76950ac700 1 -- 172.21.15.123:0/2377041048 >> 172.21.15.123:0/2377041048 conn(0x7f76900f0b70 msgr2=0x7f76900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:45.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.163+0000 7f76950ac700 1 -- 172.21.15.123:0/2377041048 shutdown_connections 2024-01-13T11:53:45.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.163+0000 7f76950ac700 1 -- 172.21.15.123:0/2377041048 wait complete. 2024-01-13T11:53:45.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.164+0000 7f76950ac700 1 Processor -- start 2024-01-13T11:53:45.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.164+0000 7f76950ac700 1 -- start start 2024-01-13T11:53:45.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.164+0000 7f76950ac700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76900601e0 0x7f76901066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:45.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.164+0000 7f76950ac700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7690106bd0 con 0x7f76900601e0 2024-01-13T11:53:45.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.164+0000 7f768ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76900601e0 0x7f76901066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:45.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.164+0000 7f768ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76900601e0 0x7f76901066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:49688/0 (socket says 172.21.15.123:49688) 2024-01-13T11:53:45.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.164+0000 7f768ed9d700 1 -- 172.21.15.123:0/65745564 learned_addr learned my addr 172.21.15.123:0/65745564 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:45.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.165+0000 7f768ed9d700 1 -- 172.21.15.123:0/65745564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7678005040 con 0x7f76900601e0 2024-01-13T11:53:45.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.165+0000 7f768ed9d700 1 --2- 172.21.15.123:0/65745564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76900601e0 0x7f76901066e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f7678004dd0 tx=0x7f7678009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:45.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.165+0000 7f76877fe700 1 -- 172.21.15.123:0/65745564 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7678018070 con 0x7f76900601e0 2024-01-13T11:53:45.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.165+0000 7f76950ac700 1 -- 172.21.15.123:0/65745564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7690106dd0 con 0x7f76900601e0 2024-01-13T11:53:45.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.165+0000 7f76877fe700 1 -- 172.21.15.123:0/65745564 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7678013410 con 0x7f76900601e0 2024-01-13T11:53:45.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.165+0000 7f76877fe700 1 -- 172.21.15.123:0/65745564 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f767801c3c0 con 0x7f76900601e0 2024-01-13T11:53:45.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.165+0000 7f76950ac700 1 -- 172.21.15.123:0/65745564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7690107260 con 0x7f76900601e0 2024-01-13T11:53:45.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.166+0000 7f76877fe700 1 -- 172.21.15.123:0/65745564 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f767801d420 con 0x7f76900601e0 2024-01-13T11:53:45.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.166+0000 7f76950ac700 1 -- 172.21.15.123:0/65745564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7690045ad0 con 0x7f76900601e0 2024-01-13T11:53:45.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.166+0000 7f76877fe700 1 --2- 172.21.15.123:0/65745564 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f767c02ffb0 0x7f767c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:45.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.166+0000 7f76877fe700 1 -- 172.21.15.123:0/65745564 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f76780412e0 con 0x7f76900601e0 2024-01-13T11:53:45.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.166+0000 7f768e59c700 1 --2- 172.21.15.123:0/65745564 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f767c02ffb0 0x7f767c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:45.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.166+0000 7f768e59c700 1 --2- 172.21.15.123:0/65745564 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f767c02ffb0 0x7f767c032470 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f76800096a0 tx=0x7f7680006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:45.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.171+0000 7f76877fe700 1 -- 172.21.15.123:0/65745564 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f767801d820 con 0x7f76900601e0 2024-01-13T11:53:45.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.378+0000 7f76950ac700 1 -- 172.21.15.123:0/65745564 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76901080b0 con 0x7f76900601e0 2024-01-13T11:53:45.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.379+0000 7f76877fe700 1 -- 172.21.15.123:0/65745564 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f767801b370 con 0x7f76900601e0 2024-01-13T11:53:45.367 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:45.367 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:45.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.380+0000 7f76950ac700 1 -- 172.21.15.123:0/65745564 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f767c02ffb0 msgr2=0x7f767c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:45.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.380+0000 7f76950ac700 1 --2- 172.21.15.123:0/65745564 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f767c02ffb0 0x7f767c032470 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f76800096a0 tx=0x7f7680006b40).stop 2024-01-13T11:53:45.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.380+0000 7f76950ac700 1 -- 172.21.15.123:0/65745564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76900601e0 msgr2=0x7f76901066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:45.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.380+0000 7f76950ac700 1 --2- 172.21.15.123:0/65745564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76900601e0 0x7f76901066e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f7678004dd0 tx=0x7f7678009b30).stop 2024-01-13T11:53:45.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.380+0000 7f76950ac700 1 -- 172.21.15.123:0/65745564 shutdown_connections 2024-01-13T11:53:45.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.380+0000 7f76950ac700 1 --2- 172.21.15.123:0/65745564 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f767c02ffb0 0x7f767c032470 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:45.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.380+0000 7f76950ac700 1 --2- 172.21.15.123:0/65745564 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76900601e0 0x7f76901066e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:45.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.380+0000 7f76950ac700 1 -- 172.21.15.123:0/65745564 >> 172.21.15.123:0/65745564 conn(0x7f76900f0b70 msgr2=0x7f76900f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:45.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.380+0000 7f76950ac700 1 -- 172.21.15.123:0/65745564 shutdown_connections 2024-01-13T11:53:45.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:45.380+0000 7f76950ac700 1 -- 172.21.15.123:0/65745564 wait complete. 2024-01-13T11:53:45.370 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:45.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:45 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:43.537833+0000 mgr.smithi059.hmyldj (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:46.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:46 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:45.379870+0000 mon.smithi059 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.123:0/65745564' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:46.748 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:46.748 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:47.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:47 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:45.538105+0000 mgr.smithi059.hmyldj (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:47.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.880+0000 7f9dd3c19700 1 -- 172.21.15.123:0/1709486419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9dcc060280 msgr2=0x7f9dcc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:47.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.880+0000 7f9dd3c19700 1 --2- 172.21.15.123:0/1709486419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9dcc060280 0x7f9dcc0f90f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f9dbc009fb0 tx=0x7f9dbc009b30).stop 2024-01-13T11:53:47.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.880+0000 7f9dd3c19700 1 -- 172.21.15.123:0/1709486419 shutdown_connections 2024-01-13T11:53:47.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.880+0000 7f9dd3c19700 1 --2- 172.21.15.123:0/1709486419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9dcc060280 0x7f9dcc0f90f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:47.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.880+0000 7f9dd3c19700 1 -- 172.21.15.123:0/1709486419 >> 172.21.15.123:0/1709486419 conn(0x7f9dcc0f0b70 msgr2=0x7f9dcc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:47.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.881+0000 7f9dd3c19700 1 -- 172.21.15.123:0/1709486419 shutdown_connections 2024-01-13T11:53:47.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.881+0000 7f9dd3c19700 1 -- 172.21.15.123:0/1709486419 wait complete. 2024-01-13T11:53:47.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.881+0000 7f9dd3c19700 1 Processor -- start 2024-01-13T11:53:47.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.881+0000 7f9dd3c19700 1 -- start start 2024-01-13T11:53:47.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.881+0000 7f9dd3c19700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9dcc060280 0x7f9dcc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:47.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.881+0000 7f9dd3c19700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dcc106bd0 con 0x7f9dcc060280 2024-01-13T11:53:47.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.882+0000 7f9dd19b5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9dcc060280 0x7f9dcc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:47.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.882+0000 7f9dd19b5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9dcc060280 0x7f9dcc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:56862/0 (socket says 172.21.15.123:56862) 2024-01-13T11:53:47.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.882+0000 7f9dd19b5700 1 -- 172.21.15.123:0/1944155450 learned_addr learned my addr 172.21.15.123:0/1944155450 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:47.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.882+0000 7f9dd19b5700 1 -- 172.21.15.123:0/1944155450 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc005040 con 0x7f9dcc060280 2024-01-13T11:53:47.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.882+0000 7f9dd19b5700 1 --2- 172.21.15.123:0/1944155450 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9dcc060280 0x7f9dcc1066e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f9dbc009830 tx=0x7f9dbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:47.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.882+0000 7f9dc27fc700 1 -- 172.21.15.123:0/1944155450 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dbc009320 con 0x7f9dcc060280 2024-01-13T11:53:47.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.882+0000 7f9dd3c19700 1 -- 172.21.15.123:0/1944155450 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dcc106dd0 con 0x7f9dcc060280 2024-01-13T11:53:47.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.882+0000 7f9dc27fc700 1 -- 172.21.15.123:0/1944155450 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9dbc0136c0 con 0x7f9dcc060280 2024-01-13T11:53:47.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.882+0000 7f9dc27fc700 1 -- 172.21.15.123:0/1944155450 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dbc01ee90 con 0x7f9dcc060280 2024-01-13T11:53:47.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.883+0000 7f9dd3c19700 1 -- 172.21.15.123:0/1944155450 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dcc107260 con 0x7f9dcc060280 2024-01-13T11:53:47.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.883+0000 7f9dc27fc700 1 -- 172.21.15.123:0/1944155450 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9dbc01e420 con 0x7f9dcc060280 2024-01-13T11:53:47.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.883+0000 7f9dd3c19700 1 -- 172.21.15.123:0/1944155450 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dcc045ad0 con 0x7f9dcc060280 2024-01-13T11:53:47.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.883+0000 7f9dc27fc700 1 --2- 172.21.15.123:0/1944155450 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9db802ff10 0x7f9db80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:47.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.883+0000 7f9dc27fc700 1 -- 172.21.15.123:0/1944155450 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9dbc040f40 con 0x7f9dcc060280 2024-01-13T11:53:47.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.884+0000 7f9dd11b4700 1 --2- 172.21.15.123:0/1944155450 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9db802ff10 0x7f9db80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:47.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.884+0000 7f9dd11b4700 1 --2- 172.21.15.123:0/1944155450 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9db802ff10 0x7f9db80323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f9dc80096a0 tx=0x7f9dc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:47.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:47.888+0000 7f9dc27fc700 1 -- 172.21.15.123:0/1944155450 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9dbc06d2b0 con 0x7f9dcc060280 2024-01-13T11:53:48.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:48.098+0000 7f9dd3c19700 1 -- 172.21.15.123:0/1944155450 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9dcc1080b0 con 0x7f9dcc060280 2024-01-13T11:53:48.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:48.099+0000 7f9dc27fc700 1 -- 172.21.15.123:0/1944155450 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9dbc0095f0 con 0x7f9dcc060280 2024-01-13T11:53:48.088 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:48.088 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:48.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:48.101+0000 7f9dd3c19700 1 -- 172.21.15.123:0/1944155450 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9db802ff10 msgr2=0x7f9db80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:48.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:48.101+0000 7f9dd3c19700 1 --2- 172.21.15.123:0/1944155450 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9db802ff10 0x7f9db80323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f9dc80096a0 tx=0x7f9dc8006b40).stop 2024-01-13T11:53:48.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:48.101+0000 7f9dd3c19700 1 -- 172.21.15.123:0/1944155450 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9dcc060280 msgr2=0x7f9dcc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:48.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:48.101+0000 7f9dd3c19700 1 --2- 172.21.15.123:0/1944155450 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9dcc060280 0x7f9dcc1066e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f9dbc009830 tx=0x7f9dbc00b040).stop 2024-01-13T11:53:48.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:48.101+0000 7f9dd3c19700 1 -- 172.21.15.123:0/1944155450 shutdown_connections 2024-01-13T11:53:48.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:48.101+0000 7f9dd3c19700 1 --2- 172.21.15.123:0/1944155450 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9db802ff10 0x7f9db80323d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:48.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:48.101+0000 7f9dd3c19700 1 --2- 172.21.15.123:0/1944155450 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9dcc060280 0x7f9dcc1066e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:48.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:48.101+0000 7f9dd3c19700 1 -- 172.21.15.123:0/1944155450 >> 172.21.15.123:0/1944155450 conn(0x7f9dcc0f0b70 msgr2=0x7f9dcc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:48.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:48.101+0000 7f9dd3c19700 1 -- 172.21.15.123:0/1944155450 shutdown_connections 2024-01-13T11:53:48.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:48.101+0000 7f9dd3c19700 1 -- 172.21.15.123:0/1944155450 wait complete. 2024-01-13T11:53:48.090 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:48.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:48 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:48.100248+0000 mon.smithi059 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.123:0/1944155450' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:49.474 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:49.474 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:49.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:49 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:47.538475+0000 mgr.smithi059.hmyldj (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:50.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.672+0000 7fed9ede0700 1 -- 172.21.15.123:0/2987412332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed980f5410 msgr2=0x7fed980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.672+0000 7fed9ede0700 1 --2- 172.21.15.123:0/2987412332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed980f5410 0x7fed980f5830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fed8c009fb0 tx=0x7fed8c009b30).stop 2024-01-13T11:53:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.672+0000 7fed9ede0700 1 -- 172.21.15.123:0/2987412332 shutdown_connections 2024-01-13T11:53:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.672+0000 7fed9ede0700 1 --2- 172.21.15.123:0/2987412332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed980f5410 0x7fed980f5830 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.672+0000 7fed9ede0700 1 -- 172.21.15.123:0/2987412332 >> 172.21.15.123:0/2987412332 conn(0x7fed980f0b70 msgr2=0x7fed980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.672+0000 7fed9ede0700 1 -- 172.21.15.123:0/2987412332 shutdown_connections 2024-01-13T11:53:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.672+0000 7fed9ede0700 1 -- 172.21.15.123:0/2987412332 wait complete. 2024-01-13T11:53:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.672+0000 7fed9ede0700 1 Processor -- start 2024-01-13T11:53:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.673+0000 7fed9ede0700 1 -- start start 2024-01-13T11:53:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.673+0000 7fed9ede0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed980f5410 0x7fed98109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.673+0000 7fed9ede0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed98060240 con 0x7fed980f5410 2024-01-13T11:53:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.673+0000 7fed9cb7c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed980f5410 0x7fed98109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.673+0000 7fed9cb7c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed980f5410 0x7fed98109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:56882/0 (socket says 172.21.15.123:56882) 2024-01-13T11:53:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.673+0000 7fed9cb7c700 1 -- 172.21.15.123:0/2349824610 learned_addr learned my addr 172.21.15.123:0/2349824610 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:50.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.673+0000 7fed9cb7c700 1 -- 172.21.15.123:0/2349824610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8c005040 con 0x7fed980f5410 2024-01-13T11:53:50.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.673+0000 7fed9cb7c700 1 --2- 172.21.15.123:0/2349824610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed980f5410 0x7fed98109430 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fed8c013510 tx=0x7fed8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:50.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.674+0000 7fed957fa700 1 -- 172.21.15.123:0/2349824610 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed8c009320 con 0x7fed980f5410 2024-01-13T11:53:50.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.674+0000 7fed9ede0700 1 -- 172.21.15.123:0/2349824610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed98060440 con 0x7fed980f5410 2024-01-13T11:53:50.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.674+0000 7fed957fa700 1 -- 172.21.15.123:0/2349824610 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed8c013f00 con 0x7fed980f5410 2024-01-13T11:53:50.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.674+0000 7fed957fa700 1 -- 172.21.15.123:0/2349824610 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed8c014d90 con 0x7fed980f5410 2024-01-13T11:53:50.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.674+0000 7fed9ede0700 1 -- 172.21.15.123:0/2349824610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed98109ae0 con 0x7fed980f5410 2024-01-13T11:53:50.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.674+0000 7fed957fa700 1 -- 172.21.15.123:0/2349824610 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed8c0263a0 con 0x7fed980f5410 2024-01-13T11:53:50.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.675+0000 7fed957fa700 1 --2- 172.21.15.123:0/2349824610 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fed8802ff60 0x7fed88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:50.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.675+0000 7fed957fa700 1 -- 172.21.15.123:0/2349824610 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed8c0410a0 con 0x7fed980f5410 2024-01-13T11:53:50.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.675+0000 7fed9ede0700 1 -- 172.21.15.123:0/2349824610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed98045ad0 con 0x7fed980f5410 2024-01-13T11:53:50.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.675+0000 7fed97fff700 1 --2- 172.21.15.123:0/2349824610 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fed8802ff60 0x7fed88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:50.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.675+0000 7fed97fff700 1 --2- 172.21.15.123:0/2349824610 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fed8802ff60 0x7fed88032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fed840096a0 tx=0x7fed84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:50.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.679+0000 7fed957fa700 1 -- 172.21.15.123:0/2349824610 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed8c012920 con 0x7fed980f5410 2024-01-13T11:53:50.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.887+0000 7fed9ede0700 1 -- 172.21.15.123:0/2349824610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed9810a1c0 con 0x7fed980f5410 2024-01-13T11:53:50.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.887+0000 7fed957fa700 1 -- 172.21.15.123:0/2349824610 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fed8c02b020 con 0x7fed980f5410 2024-01-13T11:53:50.876 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:50.876 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:50.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.889+0000 7fed9ede0700 1 -- 172.21.15.123:0/2349824610 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fed8802ff60 msgr2=0x7fed88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:50.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.889+0000 7fed9ede0700 1 --2- 172.21.15.123:0/2349824610 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fed8802ff60 0x7fed88032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fed840096a0 tx=0x7fed84006b40).stop 2024-01-13T11:53:50.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.889+0000 7fed9ede0700 1 -- 172.21.15.123:0/2349824610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed980f5410 msgr2=0x7fed98109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:50.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.889+0000 7fed9ede0700 1 --2- 172.21.15.123:0/2349824610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed980f5410 0x7fed98109430 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fed8c013510 tx=0x7fed8c00b040).stop 2024-01-13T11:53:50.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.889+0000 7fed9ede0700 1 -- 172.21.15.123:0/2349824610 shutdown_connections 2024-01-13T11:53:50.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.889+0000 7fed9ede0700 1 --2- 172.21.15.123:0/2349824610 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fed8802ff60 0x7fed88032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:50.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.889+0000 7fed9ede0700 1 --2- 172.21.15.123:0/2349824610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed980f5410 0x7fed98109430 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:50.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.889+0000 7fed9ede0700 1 -- 172.21.15.123:0/2349824610 >> 172.21.15.123:0/2349824610 conn(0x7fed980f0b70 msgr2=0x7fed980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:50.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.889+0000 7fed9ede0700 1 -- 172.21.15.123:0/2349824610 shutdown_connections 2024-01-13T11:53:50.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:50.889+0000 7fed9ede0700 1 -- 172.21.15.123:0/2349824610 wait complete. 2024-01-13T11:53:50.878 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:51.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:49.538840+0000 mgr.smithi059.hmyldj (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:51.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:50.888734+0000 mon.smithi059 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.123:0/2349824610' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:52.248 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:52.248 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:53.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.380+0000 7fec797ef700 1 -- 172.21.15.123:0/2265931084 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec740f5410 msgr2=0x7fec740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:53.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.380+0000 7fec797ef700 1 --2- 172.21.15.123:0/2265931084 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec740f5410 0x7fec740f5830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fec68009fb0 tx=0x7fec68009b30).stop 2024-01-13T11:53:53.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.380+0000 7fec797ef700 1 -- 172.21.15.123:0/2265931084 shutdown_connections 2024-01-13T11:53:53.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.380+0000 7fec797ef700 1 --2- 172.21.15.123:0/2265931084 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec740f5410 0x7fec740f5830 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:53.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.380+0000 7fec797ef700 1 -- 172.21.15.123:0/2265931084 >> 172.21.15.123:0/2265931084 conn(0x7fec740f0b70 msgr2=0x7fec740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:53.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.380+0000 7fec797ef700 1 -- 172.21.15.123:0/2265931084 shutdown_connections 2024-01-13T11:53:53.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.380+0000 7fec797ef700 1 -- 172.21.15.123:0/2265931084 wait complete. 2024-01-13T11:53:53.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.381+0000 7fec797ef700 1 Processor -- start 2024-01-13T11:53:53.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.381+0000 7fec797ef700 1 -- start start 2024-01-13T11:53:53.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.381+0000 7fec797ef700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec740f5410 0x7fec74109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:53.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.381+0000 7fec797ef700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec74060220 con 0x7fec740f5410 2024-01-13T11:53:53.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.381+0000 7fec72ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec740f5410 0x7fec74109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:53.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.381+0000 7fec72ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec740f5410 0x7fec74109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:56898/0 (socket says 172.21.15.123:56898) 2024-01-13T11:53:53.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.381+0000 7fec72ffd700 1 -- 172.21.15.123:0/1043467365 learned_addr learned my addr 172.21.15.123:0/1043467365 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:53.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.382+0000 7fec72ffd700 1 -- 172.21.15.123:0/1043467365 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec68005040 con 0x7fec740f5410 2024-01-13T11:53:53.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.382+0000 7fec72ffd700 1 --2- 172.21.15.123:0/1043467365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec740f5410 0x7fec74109450 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fec68013510 tx=0x7fec6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:53.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.382+0000 7fec5b7fe700 1 -- 172.21.15.123:0/1043467365 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec68009320 con 0x7fec740f5410 2024-01-13T11:53:53.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.382+0000 7fec797ef700 1 -- 172.21.15.123:0/1043467365 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec74060420 con 0x7fec740f5410 2024-01-13T11:53:53.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.382+0000 7fec5b7fe700 1 -- 172.21.15.123:0/1043467365 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec68013f00 con 0x7fec740f5410 2024-01-13T11:53:53.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.382+0000 7fec5b7fe700 1 -- 172.21.15.123:0/1043467365 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec68014d90 con 0x7fec740f5410 2024-01-13T11:53:53.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.382+0000 7fec797ef700 1 -- 172.21.15.123:0/1043467365 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec74109b00 con 0x7fec740f5410 2024-01-13T11:53:53.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.383+0000 7fec5b7fe700 1 -- 172.21.15.123:0/1043467365 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec680095f0 con 0x7fec740f5410 2024-01-13T11:53:53.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.383+0000 7fec797ef700 1 -- 172.21.15.123:0/1043467365 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec74059100 con 0x7fec740f5410 2024-01-13T11:53:53.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.383+0000 7fec5b7fe700 1 --2- 172.21.15.123:0/1043467365 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fec5c02ff60 0x7fec5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:53.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.383+0000 7fec5b7fe700 1 -- 172.21.15.123:0/1043467365 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec68040fe0 con 0x7fec740f5410 2024-01-13T11:53:53.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.383+0000 7fec727fc700 1 --2- 172.21.15.123:0/1043467365 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fec5c02ff60 0x7fec5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:53.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.383+0000 7fec727fc700 1 --2- 172.21.15.123:0/1043467365 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fec5c02ff60 0x7fec5c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fec640096a0 tx=0x7fec64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:53.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.388+0000 7fec5b7fe700 1 -- 172.21.15.123:0/1043467365 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec68014420 con 0x7fec740f5410 2024-01-13T11:53:53.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.593+0000 7fec797ef700 1 -- 172.21.15.123:0/1043467365 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec74045ad0 con 0x7fec740f5410 2024-01-13T11:53:53.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.594+0000 7fec5b7fe700 1 -- 172.21.15.123:0/1043467365 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fec68010230 con 0x7fec740f5410 2024-01-13T11:53:53.583 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:53.583 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:53.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.596+0000 7fec797ef700 1 -- 172.21.15.123:0/1043467365 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fec5c02ff60 msgr2=0x7fec5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:53.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.596+0000 7fec797ef700 1 --2- 172.21.15.123:0/1043467365 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fec5c02ff60 0x7fec5c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fec640096a0 tx=0x7fec64006b40).stop 2024-01-13T11:53:53.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.596+0000 7fec797ef700 1 -- 172.21.15.123:0/1043467365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec740f5410 msgr2=0x7fec74109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:53.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.596+0000 7fec797ef700 1 --2- 172.21.15.123:0/1043467365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec740f5410 0x7fec74109450 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fec68013510 tx=0x7fec6800b040).stop 2024-01-13T11:53:53.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.596+0000 7fec797ef700 1 -- 172.21.15.123:0/1043467365 shutdown_connections 2024-01-13T11:53:53.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.596+0000 7fec797ef700 1 --2- 172.21.15.123:0/1043467365 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fec5c02ff60 0x7fec5c032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:53.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.596+0000 7fec797ef700 1 --2- 172.21.15.123:0/1043467365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec740f5410 0x7fec74109450 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:53.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.596+0000 7fec797ef700 1 -- 172.21.15.123:0/1043467365 >> 172.21.15.123:0/1043467365 conn(0x7fec740f0b70 msgr2=0x7fec740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:53.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.596+0000 7fec797ef700 1 -- 172.21.15.123:0/1043467365 shutdown_connections 2024-01-13T11:53:53.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:53.596+0000 7fec797ef700 1 -- 172.21.15.123:0/1043467365 wait complete. 2024-01-13T11:53:53.586 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:53.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:53 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:51.539204+0000 mgr.smithi059.hmyldj (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:54.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:54 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:53.595561+0000 mon.smithi059 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.123:0/1043467365' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:54.958 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:54.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:55.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:55 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:53.539601+0000 mgr.smithi059.hmyldj (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:56.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.090+0000 7f64b4f80700 1 -- 172.21.15.123:0/3640238207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64b00f51a0 msgr2=0x7f64b00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:56.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.090+0000 7f64b4f80700 1 --2- 172.21.15.123:0/3640238207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64b00f51a0 0x7f64b00f55c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f64a4009fb0 tx=0x7f64a4009b30).stop 2024-01-13T11:53:56.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.090+0000 7f64b4f80700 1 -- 172.21.15.123:0/3640238207 shutdown_connections 2024-01-13T11:53:56.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.090+0000 7f64b4f80700 1 --2- 172.21.15.123:0/3640238207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64b00f51a0 0x7f64b00f55c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:56.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.090+0000 7f64b4f80700 1 -- 172.21.15.123:0/3640238207 >> 172.21.15.123:0/3640238207 conn(0x7f64b00f09a0 msgr2=0x7f64b00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:56.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.090+0000 7f64b4f80700 1 -- 172.21.15.123:0/3640238207 shutdown_connections 2024-01-13T11:53:56.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.090+0000 7f64b4f80700 1 -- 172.21.15.123:0/3640238207 wait complete. 2024-01-13T11:53:56.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.090+0000 7f64b4f80700 1 Processor -- start 2024-01-13T11:53:56.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.091+0000 7f64b4f80700 1 -- start start 2024-01-13T11:53:56.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.091+0000 7f64b4f80700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64b00f51a0 0x7f64b01091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:56.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.091+0000 7f64b4f80700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b0060240 con 0x7f64b00f51a0 2024-01-13T11:53:56.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.091+0000 7f64af7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64b00f51a0 0x7f64b01091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:56.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.091+0000 7f64af7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64b00f51a0 0x7f64b01091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:56920/0 (socket says 172.21.15.123:56920) 2024-01-13T11:53:56.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.091+0000 7f64af7fe700 1 -- 172.21.15.123:0/2624359197 learned_addr learned my addr 172.21.15.123:0/2624359197 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:56.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.091+0000 7f64af7fe700 1 -- 172.21.15.123:0/2624359197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a4005040 con 0x7f64b00f51a0 2024-01-13T11:53:56.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.092+0000 7f64af7fe700 1 --2- 172.21.15.123:0/2624359197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64b00f51a0 0x7f64b01091b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f64a4013510 tx=0x7f64a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:56.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.092+0000 7f6497fff700 1 -- 172.21.15.123:0/2624359197 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64a4009320 con 0x7f64b00f51a0 2024-01-13T11:53:56.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.092+0000 7f64b4f80700 1 -- 172.21.15.123:0/2624359197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b0060440 con 0x7f64b00f51a0 2024-01-13T11:53:56.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.092+0000 7f6497fff700 1 -- 172.21.15.123:0/2624359197 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64a4013f00 con 0x7f64b00f51a0 2024-01-13T11:53:56.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.092+0000 7f6497fff700 1 -- 172.21.15.123:0/2624359197 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64a4014dc0 con 0x7f64b00f51a0 2024-01-13T11:53:56.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.092+0000 7f64b4f80700 1 -- 172.21.15.123:0/2624359197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b0109860 con 0x7f64b00f51a0 2024-01-13T11:53:56.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.093+0000 7f6497fff700 1 -- 172.21.15.123:0/2624359197 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64a401d3d0 con 0x7f64b00f51a0 2024-01-13T11:53:56.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.093+0000 7f6497fff700 1 --2- 172.21.15.123:0/2624359197 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f649802ff60 0x7f6498032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:56.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.093+0000 7f6497fff700 1 -- 172.21.15.123:0/2624359197 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64a4041020 con 0x7f64b00f51a0 2024-01-13T11:53:56.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.093+0000 7f64b4f80700 1 -- 172.21.15.123:0/2624359197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b0045ad0 con 0x7f64b00f51a0 2024-01-13T11:53:56.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.093+0000 7f64aeffd700 1 --2- 172.21.15.123:0/2624359197 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f649802ff60 0x7f6498032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:56.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.093+0000 7f64aeffd700 1 --2- 172.21.15.123:0/2624359197 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f649802ff60 0x7f6498032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f64a00096a0 tx=0x7f64a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:56.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.099+0000 7f6497fff700 1 -- 172.21.15.123:0/2624359197 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64a4012920 con 0x7f64b00f51a0 2024-01-13T11:53:56.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.305+0000 7f64b4f80700 1 -- 172.21.15.123:0/2624359197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64b00fabe0 con 0x7f64b00f51a0 2024-01-13T11:53:56.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.305+0000 7f6497fff700 1 -- 172.21.15.123:0/2624359197 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f64a402b020 con 0x7f64b00f51a0 2024-01-13T11:53:56.294 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:56.294 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:56.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.307+0000 7f64b4f80700 1 -- 172.21.15.123:0/2624359197 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f649802ff60 msgr2=0x7f6498032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:56.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.307+0000 7f64b4f80700 1 --2- 172.21.15.123:0/2624359197 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f649802ff60 0x7f6498032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f64a00096a0 tx=0x7f64a0006b40).stop 2024-01-13T11:53:56.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.307+0000 7f64b4f80700 1 -- 172.21.15.123:0/2624359197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64b00f51a0 msgr2=0x7f64b01091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:56.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.307+0000 7f64b4f80700 1 --2- 172.21.15.123:0/2624359197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64b00f51a0 0x7f64b01091b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f64a4013510 tx=0x7f64a400b040).stop 2024-01-13T11:53:56.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.307+0000 7f64b4f80700 1 -- 172.21.15.123:0/2624359197 shutdown_connections 2024-01-13T11:53:56.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.307+0000 7f64b4f80700 1 --2- 172.21.15.123:0/2624359197 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f649802ff60 0x7f6498032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:56.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.307+0000 7f64b4f80700 1 --2- 172.21.15.123:0/2624359197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64b00f51a0 0x7f64b01091b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:56.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.307+0000 7f64b4f80700 1 -- 172.21.15.123:0/2624359197 >> 172.21.15.123:0/2624359197 conn(0x7f64b00f09a0 msgr2=0x7f64b00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:56.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.307+0000 7f64b4f80700 1 -- 172.21.15.123:0/2624359197 shutdown_connections 2024-01-13T11:53:56.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:56.307+0000 7f64b4f80700 1 -- 172.21.15.123:0/2624359197 wait complete. 2024-01-13T11:53:56.296 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:57.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:55.539891+0000 mgr.smithi059.hmyldj (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:57.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:56.306704+0000 mon.smithi059 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.123:0/2624359197' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:53:57.679 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:53:57.679 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:53:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.763+0000 7f5beb59e700 1 -- 172.21.15.123:0/4163163519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bec0f51a0 msgr2=0x7f5bec0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.763+0000 7f5beb59e700 1 --2- 172.21.15.123:0/4163163519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bec0f51a0 0x7f5bec0f55c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f5bdc009fb0 tx=0x7f5bdc009b30).stop 2024-01-13T11:53:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.763+0000 7f5beb59e700 1 -- 172.21.15.123:0/4163163519 shutdown_connections 2024-01-13T11:53:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.763+0000 7f5beb59e700 1 --2- 172.21.15.123:0/4163163519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bec0f51a0 0x7f5bec0f55c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.763+0000 7f5beb59e700 1 -- 172.21.15.123:0/4163163519 >> 172.21.15.123:0/4163163519 conn(0x7f5bec0f09a0 msgr2=0x7f5bec0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.763+0000 7f5beb59e700 1 -- 172.21.15.123:0/4163163519 shutdown_connections 2024-01-13T11:53:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.763+0000 7f5beb59e700 1 -- 172.21.15.123:0/4163163519 wait complete. 2024-01-13T11:53:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.763+0000 7f5beb59e700 1 Processor -- start 2024-01-13T11:53:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.763+0000 7f5beb59e700 1 -- start start 2024-01-13T11:53:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.763+0000 7f5beb59e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bec0f51a0 0x7f5bec10b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.763+0000 7f5beb59e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec106fc0 con 0x7f5bec0f51a0 2024-01-13T11:53:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.764+0000 7f5bea59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bec0f51a0 0x7f5bec10b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.764+0000 7f5bea59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bec0f51a0 0x7f5bec10b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:54254/0 (socket says 172.21.15.123:54254) 2024-01-13T11:53:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.764+0000 7f5bea59c700 1 -- 172.21.15.123:0/2318768582 learned_addr learned my addr 172.21.15.123:0/2318768582 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:53:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.764+0000 7f5bea59c700 1 -- 172.21.15.123:0/2318768582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bdc005040 con 0x7f5bec0f51a0 2024-01-13T11:53:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.764+0000 7f5bea59c700 1 --2- 172.21.15.123:0/2318768582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bec0f51a0 0x7f5bec10b5d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f5bdc013510 tx=0x7f5bdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.764+0000 7f5bdaffd700 1 -- 172.21.15.123:0/2318768582 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bdc009320 con 0x7f5bec0f51a0 2024-01-13T11:53:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.764+0000 7f5beb59e700 1 -- 172.21.15.123:0/2318768582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bec060220 con 0x7f5bec0f51a0 2024-01-13T11:53:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.764+0000 7f5bdaffd700 1 -- 172.21.15.123:0/2318768582 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5bdc013f00 con 0x7f5bec0f51a0 2024-01-13T11:53:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.764+0000 7f5bdaffd700 1 -- 172.21.15.123:0/2318768582 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bdc014dc0 con 0x7f5bec0f51a0 2024-01-13T11:53:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.764+0000 7f5beb59e700 1 -- 172.21.15.123:0/2318768582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bec10bac0 con 0x7f5bec0f51a0 2024-01-13T11:53:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.765+0000 7f5bdaffd700 1 -- 172.21.15.123:0/2318768582 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5bdc0095f0 con 0x7f5bec0f51a0 2024-01-13T11:53:58.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.765+0000 7f5beb59e700 1 -- 172.21.15.123:0/2318768582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bec045ad0 con 0x7f5bec0f51a0 2024-01-13T11:53:58.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.765+0000 7f5bdaffd700 1 --2- 172.21.15.123:0/2318768582 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5bd402ff60 0x7f5bd4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:53:58.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.765+0000 7f5bdaffd700 1 -- 172.21.15.123:0/2318768582 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5bdc041000 con 0x7f5bec0f51a0 2024-01-13T11:53:58.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.765+0000 7f5be9d9b700 1 --2- 172.21.15.123:0/2318768582 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5bd402ff60 0x7f5bd4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:53:58.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.766+0000 7f5be9d9b700 1 --2- 172.21.15.123:0/2318768582 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5bd402ff60 0x7f5bd4032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f5be00096a0 tx=0x7f5be0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:53:58.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.770+0000 7f5bdaffd700 1 -- 172.21.15.123:0/2318768582 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5bdc014420 con 0x7f5bec0f51a0 2024-01-13T11:53:58.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.976+0000 7f5beb59e700 1 -- 172.21.15.123:0/2318768582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5bec10c280 con 0x7f5bec0f51a0 2024-01-13T11:53:58.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.976+0000 7f5bdaffd700 1 -- 172.21.15.123:0/2318768582 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5bdc010230 con 0x7f5bec0f51a0 2024-01-13T11:53:58.964 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:53:58.964 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:53:58.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.978+0000 7f5beb59e700 1 -- 172.21.15.123:0/2318768582 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5bd402ff60 msgr2=0x7f5bd4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:58.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.978+0000 7f5beb59e700 1 --2- 172.21.15.123:0/2318768582 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5bd402ff60 0x7f5bd4032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f5be00096a0 tx=0x7f5be0006b40).stop 2024-01-13T11:53:58.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.978+0000 7f5beb59e700 1 -- 172.21.15.123:0/2318768582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bec0f51a0 msgr2=0x7f5bec10b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:53:58.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.978+0000 7f5beb59e700 1 --2- 172.21.15.123:0/2318768582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bec0f51a0 0x7f5bec10b5d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f5bdc013510 tx=0x7f5bdc00b040).stop 2024-01-13T11:53:58.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.978+0000 7f5beb59e700 1 -- 172.21.15.123:0/2318768582 shutdown_connections 2024-01-13T11:53:58.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.978+0000 7f5beb59e700 1 --2- 172.21.15.123:0/2318768582 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5bd402ff60 0x7f5bd4032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:58.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.978+0000 7f5beb59e700 1 --2- 172.21.15.123:0/2318768582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bec0f51a0 0x7f5bec10b5d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:53:58.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.978+0000 7f5beb59e700 1 -- 172.21.15.123:0/2318768582 >> 172.21.15.123:0/2318768582 conn(0x7f5bec0f09a0 msgr2=0x7f5bec0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:53:58.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.978+0000 7f5beb59e700 1 -- 172.21.15.123:0/2318768582 shutdown_connections 2024-01-13T11:53:58.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:53:58.978+0000 7f5beb59e700 1 -- 172.21.15.123:0/2318768582 wait complete. 2024-01-13T11:53:58.967 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:53:59.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:59 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:57.540147+0000 mgr.smithi059.hmyldj (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:59.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:53:59 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:53:58.976773+0000 mon.smithi059 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.123:0/2318768582' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:00.359 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:00.359 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:01.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.488+0000 7fe913d31700 1 -- 172.21.15.123:0/1916161237 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe90c0f6510 msgr2=0x7fe90c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:01.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.488+0000 7fe913d31700 1 --2- 172.21.15.123:0/1916161237 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe90c0f6510 0x7fe90c0f6930 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fe8fc009fb0 tx=0x7fe8fc009b30).stop 2024-01-13T11:54:01.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.488+0000 7fe913d31700 1 -- 172.21.15.123:0/1916161237 shutdown_connections 2024-01-13T11:54:01.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.488+0000 7fe913d31700 1 --2- 172.21.15.123:0/1916161237 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe90c0f6510 0x7fe90c0f6930 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:01.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.488+0000 7fe913d31700 1 -- 172.21.15.123:0/1916161237 >> 172.21.15.123:0/1916161237 conn(0x7fe90c0f1c70 msgr2=0x7fe90c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:01.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.488+0000 7fe913d31700 1 -- 172.21.15.123:0/1916161237 shutdown_connections 2024-01-13T11:54:01.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.488+0000 7fe913d31700 1 -- 172.21.15.123:0/1916161237 wait complete. 2024-01-13T11:54:01.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.489+0000 7fe913d31700 1 Processor -- start 2024-01-13T11:54:01.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.489+0000 7fe913d31700 1 -- start start 2024-01-13T11:54:01.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.489+0000 7fe913d31700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe90c0f6510 0x7fe90c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:01.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.489+0000 7fe913d31700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c106c20 con 0x7fe90c0f6510 2024-01-13T11:54:01.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.490+0000 7fe911acd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe90c0f6510 0x7fe90c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:01.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.490+0000 7fe911acd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe90c0f6510 0x7fe90c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:54268/0 (socket says 172.21.15.123:54268) 2024-01-13T11:54:01.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.490+0000 7fe911acd700 1 -- 172.21.15.123:0/1227890711 learned_addr learned my addr 172.21.15.123:0/1227890711 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:01.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.490+0000 7fe911acd700 1 -- 172.21.15.123:0/1227890711 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8fc005040 con 0x7fe90c0f6510 2024-01-13T11:54:01.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.490+0000 7fe911acd700 1 --2- 172.21.15.123:0/1227890711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe90c0f6510 0x7fe90c106730 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fe8fc014000 tx=0x7fe8fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:01.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.490+0000 7fe9027fc700 1 -- 172.21.15.123:0/1227890711 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8fc0095f0 con 0x7fe90c0f6510 2024-01-13T11:54:01.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.490+0000 7fe913d31700 1 -- 172.21.15.123:0/1227890711 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe90c106e20 con 0x7fe90c0f6510 2024-01-13T11:54:01.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.490+0000 7fe9027fc700 1 -- 172.21.15.123:0/1227890711 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8fc007e70 con 0x7fe90c0f6510 2024-01-13T11:54:01.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.490+0000 7fe913d31700 1 -- 172.21.15.123:0/1227890711 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90c1072b0 con 0x7fe90c0f6510 2024-01-13T11:54:01.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.490+0000 7fe9027fc700 1 -- 172.21.15.123:0/1227890711 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8fc014ea0 con 0x7fe90c0f6510 2024-01-13T11:54:01.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.491+0000 7fe9027fc700 1 -- 172.21.15.123:0/1227890711 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8fc025070 con 0x7fe90c0f6510 2024-01-13T11:54:01.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.491+0000 7fe913d31700 1 -- 172.21.15.123:0/1227890711 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe90c045ad0 con 0x7fe90c0f6510 2024-01-13T11:54:01.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.491+0000 7fe9027fc700 1 --2- 172.21.15.123:0/1227890711 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe8f802ff60 0x7fe8f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:01.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.492+0000 7fe9027fc700 1 -- 172.21.15.123:0/1227890711 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe8fc040940 con 0x7fe90c0f6510 2024-01-13T11:54:01.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.492+0000 7fe9112cc700 1 --2- 172.21.15.123:0/1227890711 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe8f802ff60 0x7fe8f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:01.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.492+0000 7fe9112cc700 1 --2- 172.21.15.123:0/1227890711 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe8f802ff60 0x7fe8f8032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fe9080096a0 tx=0x7fe908006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:01.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.497+0000 7fe9027fc700 1 -- 172.21.15.123:0/1227890711 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe8fc0079c0 con 0x7fe90c0f6510 2024-01-13T11:54:01.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:01 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:53:59.540523+0000 mgr.smithi059.hmyldj (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:01.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.703+0000 7fe913d31700 1 -- 172.21.15.123:0/1227890711 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe90c0fbf60 con 0x7fe90c0f6510 2024-01-13T11:54:01.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.704+0000 7fe9027fc700 1 -- 172.21.15.123:0/1227890711 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe8fc00f300 con 0x7fe90c0f6510 2024-01-13T11:54:01.692 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:01.692 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:01.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.706+0000 7fe913d31700 1 -- 172.21.15.123:0/1227890711 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe8f802ff60 msgr2=0x7fe8f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:01.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.706+0000 7fe913d31700 1 --2- 172.21.15.123:0/1227890711 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe8f802ff60 0x7fe8f8032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fe9080096a0 tx=0x7fe908006b40).stop 2024-01-13T11:54:01.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.706+0000 7fe913d31700 1 -- 172.21.15.123:0/1227890711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe90c0f6510 msgr2=0x7fe90c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:01.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.706+0000 7fe913d31700 1 --2- 172.21.15.123:0/1227890711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe90c0f6510 0x7fe90c106730 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fe8fc014000 tx=0x7fe8fc00b040).stop 2024-01-13T11:54:01.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.706+0000 7fe913d31700 1 -- 172.21.15.123:0/1227890711 shutdown_connections 2024-01-13T11:54:01.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.706+0000 7fe913d31700 1 --2- 172.21.15.123:0/1227890711 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe8f802ff60 0x7fe8f8032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:01.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.706+0000 7fe913d31700 1 --2- 172.21.15.123:0/1227890711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe90c0f6510 0x7fe90c106730 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:01.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.706+0000 7fe913d31700 1 -- 172.21.15.123:0/1227890711 >> 172.21.15.123:0/1227890711 conn(0x7fe90c0f1c70 msgr2=0x7fe90c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:01.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.706+0000 7fe913d31700 1 -- 172.21.15.123:0/1227890711 shutdown_connections 2024-01-13T11:54:01.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:01.706+0000 7fe913d31700 1 -- 172.21.15.123:0/1227890711 wait complete. 2024-01-13T11:54:01.695 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:02.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:02 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:01.704237+0000 mon.smithi059 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.123:0/1227890711' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:03.022 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:03.023 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:03.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:03 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:01.540881+0000 mgr.smithi059.hmyldj (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:04.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.095+0000 7f021203f700 1 -- 172.21.15.123:0/1099261727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f020c0f51a0 msgr2=0x7f020c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:04.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.095+0000 7f021203f700 1 --2- 172.21.15.123:0/1099261727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f020c0f51a0 0x7f020c0f55c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f0208009fb0 tx=0x7f0208009b30).stop 2024-01-13T11:54:04.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.095+0000 7f021203f700 1 -- 172.21.15.123:0/1099261727 shutdown_connections 2024-01-13T11:54:04.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.095+0000 7f021203f700 1 --2- 172.21.15.123:0/1099261727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f020c0f51a0 0x7f020c0f55c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:04.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.095+0000 7f021203f700 1 -- 172.21.15.123:0/1099261727 >> 172.21.15.123:0/1099261727 conn(0x7f020c0f09a0 msgr2=0x7f020c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:04.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.095+0000 7f021203f700 1 -- 172.21.15.123:0/1099261727 shutdown_connections 2024-01-13T11:54:04.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.095+0000 7f021203f700 1 -- 172.21.15.123:0/1099261727 wait complete. 2024-01-13T11:54:04.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.095+0000 7f021203f700 1 Processor -- start 2024-01-13T11:54:04.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.095+0000 7f021203f700 1 -- start start 2024-01-13T11:54:04.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.095+0000 7f021203f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f020c0f51a0 0x7f020c10b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:04.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.095+0000 7f021203f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020c106fc0 con 0x7f020c0f51a0 2024-01-13T11:54:04.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.096+0000 7f021103d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f020c0f51a0 0x7f020c10b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:04.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.096+0000 7f021103d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f020c0f51a0 0x7f020c10b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:54288/0 (socket says 172.21.15.123:54288) 2024-01-13T11:54:04.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.096+0000 7f021103d700 1 -- 172.21.15.123:0/1778179056 learned_addr learned my addr 172.21.15.123:0/1778179056 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:04.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.096+0000 7f021103d700 1 -- 172.21.15.123:0/1778179056 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0208005040 con 0x7f020c0f51a0 2024-01-13T11:54:04.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.096+0000 7f021103d700 1 --2- 172.21.15.123:0/1778179056 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f020c0f51a0 0x7f020c10b5d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f0208013510 tx=0x7f020800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:04.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.096+0000 7f0201ffb700 1 -- 172.21.15.123:0/1778179056 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0208009320 con 0x7f020c0f51a0 2024-01-13T11:54:04.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.096+0000 7f021203f700 1 -- 172.21.15.123:0/1778179056 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f020c060220 con 0x7f020c0f51a0 2024-01-13T11:54:04.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.097+0000 7f0201ffb700 1 -- 172.21.15.123:0/1778179056 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0208013f00 con 0x7f020c0f51a0 2024-01-13T11:54:04.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.097+0000 7f0201ffb700 1 -- 172.21.15.123:0/1778179056 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0208014dc0 con 0x7f020c0f51a0 2024-01-13T11:54:04.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.097+0000 7f021203f700 1 -- 172.21.15.123:0/1778179056 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f020c10bac0 con 0x7f020c0f51a0 2024-01-13T11:54:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.097+0000 7f0201ffb700 1 -- 172.21.15.123:0/1778179056 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02080095f0 con 0x7f020c0f51a0 2024-01-13T11:54:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.097+0000 7f021203f700 1 -- 172.21.15.123:0/1778179056 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f020c045ad0 con 0x7f020c0f51a0 2024-01-13T11:54:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.098+0000 7f0201ffb700 1 --2- 172.21.15.123:0/1778179056 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f01f802ff60 0x7f01f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.098+0000 7f0201ffb700 1 -- 172.21.15.123:0/1778179056 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0208041020 con 0x7f020c0f51a0 2024-01-13T11:54:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.098+0000 7f021083c700 1 --2- 172.21.15.123:0/1778179056 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f01f802ff60 0x7f01f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.098+0000 7f021083c700 1 --2- 172.21.15.123:0/1778179056 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f01f802ff60 0x7f01f8032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f01fc0096a0 tx=0x7f01fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:04.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.102+0000 7f0201ffb700 1 -- 172.21.15.123:0/1778179056 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0208014420 con 0x7f020c0f51a0 2024-01-13T11:54:04.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.311+0000 7f021203f700 1 -- 172.21.15.123:0/1778179056 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f020c10c2b0 con 0x7f020c0f51a0 2024-01-13T11:54:04.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.312+0000 7f0201ffb700 1 -- 172.21.15.123:0/1778179056 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f020801c920 con 0x7f020c0f51a0 2024-01-13T11:54:04.300 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:04.300 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:04.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.313+0000 7f021203f700 1 -- 172.21.15.123:0/1778179056 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f01f802ff60 msgr2=0x7f01f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:04.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.313+0000 7f021203f700 1 --2- 172.21.15.123:0/1778179056 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f01f802ff60 0x7f01f8032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f01fc0096a0 tx=0x7f01fc006b40).stop 2024-01-13T11:54:04.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.313+0000 7f021203f700 1 -- 172.21.15.123:0/1778179056 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f020c0f51a0 msgr2=0x7f020c10b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:04.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.313+0000 7f021203f700 1 --2- 172.21.15.123:0/1778179056 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f020c0f51a0 0x7f020c10b5d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f0208013510 tx=0x7f020800b040).stop 2024-01-13T11:54:04.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.313+0000 7f021203f700 1 -- 172.21.15.123:0/1778179056 shutdown_connections 2024-01-13T11:54:04.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.313+0000 7f021203f700 1 --2- 172.21.15.123:0/1778179056 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f01f802ff60 0x7f01f8032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:04.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.313+0000 7f021203f700 1 --2- 172.21.15.123:0/1778179056 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f020c0f51a0 0x7f020c10b5d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:04.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.313+0000 7f021203f700 1 -- 172.21.15.123:0/1778179056 >> 172.21.15.123:0/1778179056 conn(0x7f020c0f09a0 msgr2=0x7f020c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:04.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.314+0000 7f021203f700 1 -- 172.21.15.123:0/1778179056 shutdown_connections 2024-01-13T11:54:04.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:04.314+0000 7f021203f700 1 -- 172.21.15.123:0/1778179056 wait complete. 2024-01-13T11:54:04.303 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:05.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:05 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:03.541279+0000 mgr.smithi059.hmyldj (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:05.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:05 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:04.312921+0000 mon.smithi059 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.123:0/1778179056' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:05.668 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:05.669 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:06.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.797+0000 7f8c0688e700 1 -- 172.21.15.123:0/695008226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c000f78e0 msgr2=0x7f8c000f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:06.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.797+0000 7f8c0688e700 1 --2- 172.21.15.123:0/695008226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c000f78e0 0x7f8c000f7d00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f8bfc009fb0 tx=0x7f8bfc009b30).stop 2024-01-13T11:54:06.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.797+0000 7f8c0688e700 1 -- 172.21.15.123:0/695008226 shutdown_connections 2024-01-13T11:54:06.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.797+0000 7f8c0688e700 1 --2- 172.21.15.123:0/695008226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c000f78e0 0x7f8c000f7d00 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:06.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.797+0000 7f8c0688e700 1 -- 172.21.15.123:0/695008226 >> 172.21.15.123:0/695008226 conn(0x7f8c000f0120 msgr2=0x7f8c000f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:06.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.798+0000 7f8c0688e700 1 -- 172.21.15.123:0/695008226 shutdown_connections 2024-01-13T11:54:06.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.798+0000 7f8c0688e700 1 -- 172.21.15.123:0/695008226 wait complete. 2024-01-13T11:54:06.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.798+0000 7f8c0688e700 1 Processor -- start 2024-01-13T11:54:06.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.798+0000 7f8c0688e700 1 -- start start 2024-01-13T11:54:06.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.798+0000 7f8c0688e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c000f78e0 0x7f8c001064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:06.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.798+0000 7f8c0688e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c001069d0 con 0x7f8c000f78e0 2024-01-13T11:54:06.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.799+0000 7f8c0588c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c000f78e0 0x7f8c001064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:06.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.799+0000 7f8c0588c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c000f78e0 0x7f8c001064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:37114/0 (socket says 172.21.15.123:37114) 2024-01-13T11:54:06.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.799+0000 7f8c0588c700 1 -- 172.21.15.123:0/50933647 learned_addr learned my addr 172.21.15.123:0/50933647 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:06.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.799+0000 7f8c0588c700 1 -- 172.21.15.123:0/50933647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc005040 con 0x7f8c000f78e0 2024-01-13T11:54:06.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.799+0000 7f8c0588c700 1 --2- 172.21.15.123:0/50933647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c000f78e0 0x7f8c001064e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f8bfc013510 tx=0x7f8bfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:06.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.799+0000 7f8bf27fc700 1 -- 172.21.15.123:0/50933647 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bfc009320 con 0x7f8c000f78e0 2024-01-13T11:54:06.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.799+0000 7f8c0688e700 1 -- 172.21.15.123:0/50933647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c00106bd0 con 0x7f8c000f78e0 2024-01-13T11:54:06.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.799+0000 7f8bf27fc700 1 -- 172.21.15.123:0/50933647 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8bfc013f00 con 0x7f8c000f78e0 2024-01-13T11:54:06.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.799+0000 7f8bf27fc700 1 -- 172.21.15.123:0/50933647 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bfc014dc0 con 0x7f8c000f78e0 2024-01-13T11:54:06.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.799+0000 7f8c0688e700 1 -- 172.21.15.123:0/50933647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c00107060 con 0x7f8c000f78e0 2024-01-13T11:54:06.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.800+0000 7f8bf27fc700 1 -- 172.21.15.123:0/50933647 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8bfc01d3d0 con 0x7f8c000f78e0 2024-01-13T11:54:06.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.800+0000 7f8bf27fc700 1 --2- 172.21.15.123:0/50933647 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8bec02ff60 0x7f8bec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:06.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.800+0000 7f8bf27fc700 1 -- 172.21.15.123:0/50933647 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8bfc041040 con 0x7f8c000f78e0 2024-01-13T11:54:06.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.800+0000 7f8c0688e700 1 -- 172.21.15.123:0/50933647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c00045ad0 con 0x7f8c000f78e0 2024-01-13T11:54:06.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.801+0000 7f8c0508b700 1 --2- 172.21.15.123:0/50933647 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8bec02ff60 0x7f8bec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:06.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.801+0000 7f8c0508b700 1 --2- 172.21.15.123:0/50933647 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8bec02ff60 0x7f8bec032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f8bf40096a0 tx=0x7f8bf4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:06.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:06.805+0000 7f8bf27fc700 1 -- 172.21.15.123:0/50933647 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8bfc012920 con 0x7f8c000f78e0 2024-01-13T11:54:07.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:07.012+0000 7f8c0688e700 1 -- 172.21.15.123:0/50933647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c000fbdd0 con 0x7f8c000f78e0 2024-01-13T11:54:07.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:07.012+0000 7f8bf27fc700 1 -- 172.21.15.123:0/50933647 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8bfc02b020 con 0x7f8c000f78e0 2024-01-13T11:54:07.001 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:07.001 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:07.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:07.014+0000 7f8c0688e700 1 -- 172.21.15.123:0/50933647 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8bec02ff60 msgr2=0x7f8bec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:07.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:07.014+0000 7f8c0688e700 1 --2- 172.21.15.123:0/50933647 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8bec02ff60 0x7f8bec032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f8bf40096a0 tx=0x7f8bf4006b40).stop 2024-01-13T11:54:07.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:07.014+0000 7f8c0688e700 1 -- 172.21.15.123:0/50933647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c000f78e0 msgr2=0x7f8c001064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:07.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:07.014+0000 7f8c0688e700 1 --2- 172.21.15.123:0/50933647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c000f78e0 0x7f8c001064e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f8bfc013510 tx=0x7f8bfc00b040).stop 2024-01-13T11:54:07.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:07.014+0000 7f8c0688e700 1 -- 172.21.15.123:0/50933647 shutdown_connections 2024-01-13T11:54:07.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:07.014+0000 7f8c0688e700 1 --2- 172.21.15.123:0/50933647 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8bec02ff60 0x7f8bec032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:07.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:07.014+0000 7f8c0688e700 1 --2- 172.21.15.123:0/50933647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c000f78e0 0x7f8c001064e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:07.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:07.014+0000 7f8c0688e700 1 -- 172.21.15.123:0/50933647 >> 172.21.15.123:0/50933647 conn(0x7f8c000f0120 msgr2=0x7f8c000f0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:07.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:07.014+0000 7f8c0688e700 1 -- 172.21.15.123:0/50933647 shutdown_connections 2024-01-13T11:54:07.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:07.014+0000 7f8c0688e700 1 -- 172.21.15.123:0/50933647 wait complete. 2024-01-13T11:54:07.004 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:07.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:05.541704+0000 mgr.smithi059.hmyldj (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:07.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:07.013845+0000 mon.smithi059 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.123:0/50933647' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:08.396 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:08.397 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:09.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.468+0000 7f88ff583700 1 -- 172.21.15.123:0/2105601579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f80f6510 msgr2=0x7f88f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:09.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.468+0000 7f88ff583700 1 --2- 172.21.15.123:0/2105601579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f80f6510 0x7f88f80f6930 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f88ec009fb0 tx=0x7f88ec009b30).stop 2024-01-13T11:54:09.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.468+0000 7f88ff583700 1 -- 172.21.15.123:0/2105601579 shutdown_connections 2024-01-13T11:54:09.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.468+0000 7f88ff583700 1 --2- 172.21.15.123:0/2105601579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f80f6510 0x7f88f80f6930 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:09.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.468+0000 7f88ff583700 1 -- 172.21.15.123:0/2105601579 >> 172.21.15.123:0/2105601579 conn(0x7f88f80f1c70 msgr2=0x7f88f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:09.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.468+0000 7f88ff583700 1 -- 172.21.15.123:0/2105601579 shutdown_connections 2024-01-13T11:54:09.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.468+0000 7f88ff583700 1 -- 172.21.15.123:0/2105601579 wait complete. 2024-01-13T11:54:09.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.469+0000 7f88ff583700 1 Processor -- start 2024-01-13T11:54:09.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.469+0000 7f88ff583700 1 -- start start 2024-01-13T11:54:09.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.469+0000 7f88ff583700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f80f6510 0x7f88f806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:09.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.469+0000 7f88ff583700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f8069e20 con 0x7f88f80f6510 2024-01-13T11:54:09.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.469+0000 7f88fd31f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f80f6510 0x7f88f806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:09.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.469+0000 7f88fd31f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f80f6510 0x7f88f806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:37144/0 (socket says 172.21.15.123:37144) 2024-01-13T11:54:09.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.469+0000 7f88fd31f700 1 -- 172.21.15.123:0/2631587568 learned_addr learned my addr 172.21.15.123:0/2631587568 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:09.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.470+0000 7f88fd31f700 1 -- 172.21.15.123:0/2631587568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec005040 con 0x7f88f80f6510 2024-01-13T11:54:09.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.470+0000 7f88fd31f700 1 --2- 172.21.15.123:0/2631587568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f80f6510 0x7f88f806b9a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f88ec013510 tx=0x7f88ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:09.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.470+0000 7f88e9ffb700 1 -- 172.21.15.123:0/2631587568 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88ec009320 con 0x7f88f80f6510 2024-01-13T11:54:09.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.470+0000 7f88e9ffb700 1 -- 172.21.15.123:0/2631587568 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88ec013f00 con 0x7f88f80f6510 2024-01-13T11:54:09.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.470+0000 7f88ff583700 1 -- 172.21.15.123:0/2631587568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f8069fc0 con 0x7f88f80f6510 2024-01-13T11:54:09.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.470+0000 7f88e9ffb700 1 -- 172.21.15.123:0/2631587568 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88ec014d90 con 0x7f88f80f6510 2024-01-13T11:54:09.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.470+0000 7f88ff583700 1 -- 172.21.15.123:0/2631587568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f806a450 con 0x7f88f80f6510 2024-01-13T11:54:09.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.471+0000 7f88e9ffb700 1 -- 172.21.15.123:0/2631587568 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88ec0095f0 con 0x7f88f80f6510 2024-01-13T11:54:09.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.471+0000 7f88ff583700 1 -- 172.21.15.123:0/2631587568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f8059100 con 0x7f88f80f6510 2024-01-13T11:54:09.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.471+0000 7f88e9ffb700 1 --2- 172.21.15.123:0/2631587568 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f88e402ffb0 0x7f88e4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:09.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.471+0000 7f88e9ffb700 1 -- 172.21.15.123:0/2631587568 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f88ec041050 con 0x7f88f80f6510 2024-01-13T11:54:09.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.471+0000 7f88fcb1e700 1 --2- 172.21.15.123:0/2631587568 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f88e402ffb0 0x7f88e4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:09.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.472+0000 7f88fcb1e700 1 --2- 172.21.15.123:0/2631587568 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f88e402ffb0 0x7f88e4032470 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f88f40096a0 tx=0x7f88f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:09.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.476+0000 7f88e9ffb700 1 -- 172.21.15.123:0/2631587568 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f88ec01d7f0 con 0x7f88f80f6510 2024-01-13T11:54:09.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:09 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:07.542114+0000 mgr.smithi059.hmyldj (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:09.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.680+0000 7f88ff583700 1 -- 172.21.15.123:0/2631587568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88f8045ad0 con 0x7f88f80f6510 2024-01-13T11:54:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.681+0000 7f88e9ffb700 1 -- 172.21.15.123:0/2631587568 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f88ec010230 con 0x7f88f80f6510 2024-01-13T11:54:09.669 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:09.669 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:09.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.683+0000 7f88ff583700 1 -- 172.21.15.123:0/2631587568 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f88e402ffb0 msgr2=0x7f88e4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:09.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.683+0000 7f88ff583700 1 --2- 172.21.15.123:0/2631587568 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f88e402ffb0 0x7f88e4032470 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f88f40096a0 tx=0x7f88f4006b40).stop 2024-01-13T11:54:09.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.683+0000 7f88ff583700 1 -- 172.21.15.123:0/2631587568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f80f6510 msgr2=0x7f88f806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:09.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.683+0000 7f88ff583700 1 --2- 172.21.15.123:0/2631587568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f80f6510 0x7f88f806b9a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f88ec013510 tx=0x7f88ec00b040).stop 2024-01-13T11:54:09.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.683+0000 7f88ff583700 1 -- 172.21.15.123:0/2631587568 shutdown_connections 2024-01-13T11:54:09.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.683+0000 7f88ff583700 1 --2- 172.21.15.123:0/2631587568 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f88e402ffb0 0x7f88e4032470 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:09.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.683+0000 7f88ff583700 1 --2- 172.21.15.123:0/2631587568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f80f6510 0x7f88f806b9a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:09.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.683+0000 7f88ff583700 1 -- 172.21.15.123:0/2631587568 >> 172.21.15.123:0/2631587568 conn(0x7f88f80f1c70 msgr2=0x7f88f80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:09.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.683+0000 7f88ff583700 1 -- 172.21.15.123:0/2631587568 shutdown_connections 2024-01-13T11:54:09.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:09.683+0000 7f88ff583700 1 -- 172.21.15.123:0/2631587568 wait complete. 2024-01-13T11:54:09.673 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:10.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:10 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:09.682054+0000 mon.smithi059 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.123:0/2631587568' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:11.052 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:11.052 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:11.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:11 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:09.542588+0000 mgr.smithi059.hmyldj (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:12.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.221+0000 7f1e4f880700 1 -- 172.21.15.123:0/4169482314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e480f5410 msgr2=0x7f1e480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:12.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.221+0000 7f1e4f880700 1 --2- 172.21.15.123:0/4169482314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e480f5410 0x7f1e480f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f1e38009fb0 tx=0x7f1e38009b30).stop 2024-01-13T11:54:12.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.221+0000 7f1e4f880700 1 -- 172.21.15.123:0/4169482314 shutdown_connections 2024-01-13T11:54:12.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.221+0000 7f1e4f880700 1 --2- 172.21.15.123:0/4169482314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e480f5410 0x7f1e480f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:12.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.221+0000 7f1e4f880700 1 -- 172.21.15.123:0/4169482314 >> 172.21.15.123:0/4169482314 conn(0x7f1e480f0b70 msgr2=0x7f1e480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:12.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.221+0000 7f1e4f880700 1 -- 172.21.15.123:0/4169482314 shutdown_connections 2024-01-13T11:54:12.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.221+0000 7f1e4f880700 1 -- 172.21.15.123:0/4169482314 wait complete. 2024-01-13T11:54:12.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.222+0000 7f1e4f880700 1 Processor -- start 2024-01-13T11:54:12.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.222+0000 7f1e4f880700 1 -- start start 2024-01-13T11:54:12.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.222+0000 7f1e4f880700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e480f5410 0x7f1e48106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:12.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.222+0000 7f1e4f880700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e48106c10 con 0x7f1e480f5410 2024-01-13T11:54:12.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.222+0000 7f1e4d61c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e480f5410 0x7f1e48106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:12.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.222+0000 7f1e4d61c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e480f5410 0x7f1e48106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:37152/0 (socket says 172.21.15.123:37152) 2024-01-13T11:54:12.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.222+0000 7f1e4d61c700 1 -- 172.21.15.123:0/2505635703 learned_addr learned my addr 172.21.15.123:0/2505635703 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:12.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.223+0000 7f1e4d61c700 1 -- 172.21.15.123:0/2505635703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e38005040 con 0x7f1e480f5410 2024-01-13T11:54:12.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.223+0000 7f1e4d61c700 1 --2- 172.21.15.123:0/2505635703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e480f5410 0x7f1e48106720 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f1e38013510 tx=0x7f1e3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:12.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.223+0000 7f1e3dffb700 1 -- 172.21.15.123:0/2505635703 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e38009320 con 0x7f1e480f5410 2024-01-13T11:54:12.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.223+0000 7f1e4f880700 1 -- 172.21.15.123:0/2505635703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e48106e10 con 0x7f1e480f5410 2024-01-13T11:54:12.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.223+0000 7f1e3dffb700 1 -- 172.21.15.123:0/2505635703 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e38013f00 con 0x7f1e480f5410 2024-01-13T11:54:12.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.223+0000 7f1e3dffb700 1 -- 172.21.15.123:0/2505635703 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e38014d90 con 0x7f1e480f5410 2024-01-13T11:54:12.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.223+0000 7f1e4f880700 1 -- 172.21.15.123:0/2505635703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e481072a0 con 0x7f1e480f5410 2024-01-13T11:54:12.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.224+0000 7f1e3dffb700 1 -- 172.21.15.123:0/2505635703 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e380263a0 con 0x7f1e480f5410 2024-01-13T11:54:12.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.224+0000 7f1e3dffb700 1 --2- 172.21.15.123:0/2505635703 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1e3402ffb0 0x7f1e34032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:12.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.224+0000 7f1e3dffb700 1 -- 172.21.15.123:0/2505635703 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e38041090 con 0x7f1e480f5410 2024-01-13T11:54:12.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.224+0000 7f1e4f880700 1 -- 172.21.15.123:0/2505635703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e48045ad0 con 0x7f1e480f5410 2024-01-13T11:54:12.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.225+0000 7f1e4ce1b700 1 --2- 172.21.15.123:0/2505635703 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1e3402ffb0 0x7f1e34032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:12.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.225+0000 7f1e4ce1b700 1 --2- 172.21.15.123:0/2505635703 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1e3402ffb0 0x7f1e34032470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f1e440096a0 tx=0x7f1e44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:12.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.229+0000 7f1e3dffb700 1 -- 172.21.15.123:0/2505635703 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e38012920 con 0x7f1e480f5410 2024-01-13T11:54:12.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.439+0000 7f1e4f880700 1 -- 172.21.15.123:0/2505635703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e48108150 con 0x7f1e480f5410 2024-01-13T11:54:12.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.439+0000 7f1e3dffb700 1 -- 172.21.15.123:0/2505635703 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1e3802b020 con 0x7f1e480f5410 2024-01-13T11:54:12.428 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:12.428 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:12.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.441+0000 7f1e4f880700 1 -- 172.21.15.123:0/2505635703 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1e3402ffb0 msgr2=0x7f1e34032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:12.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.441+0000 7f1e4f880700 1 --2- 172.21.15.123:0/2505635703 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1e3402ffb0 0x7f1e34032470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f1e440096a0 tx=0x7f1e44006b40).stop 2024-01-13T11:54:12.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.441+0000 7f1e4f880700 1 -- 172.21.15.123:0/2505635703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e480f5410 msgr2=0x7f1e48106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:12.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.441+0000 7f1e4f880700 1 --2- 172.21.15.123:0/2505635703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e480f5410 0x7f1e48106720 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f1e38013510 tx=0x7f1e3800b040).stop 2024-01-13T11:54:12.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.441+0000 7f1e4f880700 1 -- 172.21.15.123:0/2505635703 shutdown_connections 2024-01-13T11:54:12.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.441+0000 7f1e4f880700 1 --2- 172.21.15.123:0/2505635703 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f1e3402ffb0 0x7f1e34032470 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:12.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.441+0000 7f1e4f880700 1 --2- 172.21.15.123:0/2505635703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e480f5410 0x7f1e48106720 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:12.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.441+0000 7f1e4f880700 1 -- 172.21.15.123:0/2505635703 >> 172.21.15.123:0/2505635703 conn(0x7f1e480f0b70 msgr2=0x7f1e4805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:12.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.441+0000 7f1e4f880700 1 -- 172.21.15.123:0/2505635703 shutdown_connections 2024-01-13T11:54:12.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:12.441+0000 7f1e4f880700 1 -- 172.21.15.123:0/2505635703 wait complete. 2024-01-13T11:54:12.431 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:13.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:13 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:11.542929+0000 mgr.smithi059.hmyldj (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:13.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:13 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:12.440891+0000 mon.smithi059 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.123:0/2505635703' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:13.806 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:13.806 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:14.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.952+0000 7ff996c9a700 1 -- 172.21.15.123:0/1092611439 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9900f6510 msgr2=0x7ff9900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:14.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.952+0000 7ff996c9a700 1 --2- 172.21.15.123:0/1092611439 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9900f6510 0x7ff9900f6930 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7ff984009fb0 tx=0x7ff984009b30).stop 2024-01-13T11:54:14.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.952+0000 7ff996c9a700 1 -- 172.21.15.123:0/1092611439 shutdown_connections 2024-01-13T11:54:14.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.952+0000 7ff996c9a700 1 --2- 172.21.15.123:0/1092611439 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9900f6510 0x7ff9900f6930 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:14.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.952+0000 7ff996c9a700 1 -- 172.21.15.123:0/1092611439 >> 172.21.15.123:0/1092611439 conn(0x7ff9900f1c70 msgr2=0x7ff9900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:14.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.952+0000 7ff996c9a700 1 -- 172.21.15.123:0/1092611439 shutdown_connections 2024-01-13T11:54:14.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.952+0000 7ff996c9a700 1 -- 172.21.15.123:0/1092611439 wait complete. 2024-01-13T11:54:14.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.953+0000 7ff996c9a700 1 Processor -- start 2024-01-13T11:54:14.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.953+0000 7ff996c9a700 1 -- start start 2024-01-13T11:54:14.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.953+0000 7ff996c9a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9900f6510 0x7ff9901066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:14.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.953+0000 7ff996c9a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff990106bc0 con 0x7ff9900f6510 2024-01-13T11:54:14.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.953+0000 7ff994a36700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9900f6510 0x7ff9901066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:14.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.953+0000 7ff994a36700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9900f6510 0x7ff9901066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:37172/0 (socket says 172.21.15.123:37172) 2024-01-13T11:54:14.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.953+0000 7ff994a36700 1 -- 172.21.15.123:0/4007079860 learned_addr learned my addr 172.21.15.123:0/4007079860 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:14.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.954+0000 7ff994a36700 1 -- 172.21.15.123:0/4007079860 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff984005040 con 0x7ff9900f6510 2024-01-13T11:54:14.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.954+0000 7ff994a36700 1 --2- 172.21.15.123:0/4007079860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9900f6510 0x7ff9901066d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7ff984013510 tx=0x7ff98400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:14.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.954+0000 7ff98d7fa700 1 -- 172.21.15.123:0/4007079860 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff984009320 con 0x7ff9900f6510 2024-01-13T11:54:14.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.954+0000 7ff98d7fa700 1 -- 172.21.15.123:0/4007079860 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff984013f00 con 0x7ff9900f6510 2024-01-13T11:54:14.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.954+0000 7ff996c9a700 1 -- 172.21.15.123:0/4007079860 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff990106dc0 con 0x7ff9900f6510 2024-01-13T11:54:14.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.954+0000 7ff98d7fa700 1 -- 172.21.15.123:0/4007079860 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff984014d90 con 0x7ff9900f6510 2024-01-13T11:54:14.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.954+0000 7ff996c9a700 1 -- 172.21.15.123:0/4007079860 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff990107250 con 0x7ff9900f6510 2024-01-13T11:54:14.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.955+0000 7ff98d7fa700 1 -- 172.21.15.123:0/4007079860 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff9840095f0 con 0x7ff9900f6510 2024-01-13T11:54:14.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.955+0000 7ff996c9a700 1 -- 172.21.15.123:0/4007079860 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff990045ad0 con 0x7ff9900f6510 2024-01-13T11:54:14.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.955+0000 7ff98d7fa700 1 --2- 172.21.15.123:0/4007079860 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff98002ff60 0x7ff980032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:14.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.955+0000 7ff98d7fa700 1 -- 172.21.15.123:0/4007079860 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff984041080 con 0x7ff9900f6510 2024-01-13T11:54:14.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.955+0000 7ff98ffff700 1 --2- 172.21.15.123:0/4007079860 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff98002ff60 0x7ff980032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:14.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.956+0000 7ff98ffff700 1 --2- 172.21.15.123:0/4007079860 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff98002ff60 0x7ff980032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7ff97c0096a0 tx=0x7ff97c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:14.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:14.960+0000 7ff98d7fa700 1 -- 172.21.15.123:0/4007079860 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff98401d7f0 con 0x7ff9900f6510 2024-01-13T11:54:15.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:15.166+0000 7ff996c9a700 1 -- 172.21.15.123:0/4007079860 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff990108100 con 0x7ff9900f6510 2024-01-13T11:54:15.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:15.167+0000 7ff98d7fa700 1 -- 172.21.15.123:0/4007079860 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff984010230 con 0x7ff9900f6510 2024-01-13T11:54:15.155 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:15.155 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:15.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:15.168+0000 7ff996c9a700 1 -- 172.21.15.123:0/4007079860 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff98002ff60 msgr2=0x7ff980032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:15.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:15.168+0000 7ff996c9a700 1 --2- 172.21.15.123:0/4007079860 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff98002ff60 0x7ff980032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7ff97c0096a0 tx=0x7ff97c006b40).stop 2024-01-13T11:54:15.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:15.168+0000 7ff996c9a700 1 -- 172.21.15.123:0/4007079860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9900f6510 msgr2=0x7ff9901066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:15.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:15.168+0000 7ff996c9a700 1 --2- 172.21.15.123:0/4007079860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9900f6510 0x7ff9901066d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7ff984013510 tx=0x7ff98400b040).stop 2024-01-13T11:54:15.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:15.168+0000 7ff996c9a700 1 -- 172.21.15.123:0/4007079860 shutdown_connections 2024-01-13T11:54:15.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:15.168+0000 7ff996c9a700 1 --2- 172.21.15.123:0/4007079860 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff98002ff60 0x7ff980032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:15.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:15.168+0000 7ff996c9a700 1 --2- 172.21.15.123:0/4007079860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9900f6510 0x7ff9901066d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:15.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:15.168+0000 7ff996c9a700 1 -- 172.21.15.123:0/4007079860 >> 172.21.15.123:0/4007079860 conn(0x7ff9900f1c70 msgr2=0x7ff9900f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:15.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:15.169+0000 7ff996c9a700 1 -- 172.21.15.123:0/4007079860 shutdown_connections 2024-01-13T11:54:15.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:15.169+0000 7ff996c9a700 1 -- 172.21.15.123:0/4007079860 wait complete. 2024-01-13T11:54:15.158 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:15.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:15 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:13.543274+0000 mgr.smithi059.hmyldj (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:15.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:15 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:15.168235+0000 mon.smithi059 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.123:0/4007079860' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:16.538 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:16.538 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:17.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:17 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:15.543613+0000 mgr.smithi059.hmyldj (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:17.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.694+0000 7f65eed10700 1 -- 172.21.15.123:0/2051314231 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65e80f5410 msgr2=0x7f65e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:17.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.694+0000 7f65eed10700 1 --2- 172.21.15.123:0/2051314231 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65e80f5410 0x7f65e80f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f65dc009fb0 tx=0x7f65dc009b30).stop 2024-01-13T11:54:17.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.694+0000 7f65eed10700 1 -- 172.21.15.123:0/2051314231 shutdown_connections 2024-01-13T11:54:17.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.694+0000 7f65eed10700 1 --2- 172.21.15.123:0/2051314231 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65e80f5410 0x7f65e80f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:17.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.694+0000 7f65eed10700 1 -- 172.21.15.123:0/2051314231 >> 172.21.15.123:0/2051314231 conn(0x7f65e80f0b70 msgr2=0x7f65e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:17.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.694+0000 7f65eed10700 1 -- 172.21.15.123:0/2051314231 shutdown_connections 2024-01-13T11:54:17.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.694+0000 7f65eed10700 1 -- 172.21.15.123:0/2051314231 wait complete. 2024-01-13T11:54:17.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.695+0000 7f65eed10700 1 Processor -- start 2024-01-13T11:54:17.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.695+0000 7f65eed10700 1 -- start start 2024-01-13T11:54:17.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.695+0000 7f65eed10700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65e80f5410 0x7f65e8108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:17.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.695+0000 7f65eed10700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e8108e00 con 0x7f65e80f5410 2024-01-13T11:54:17.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.695+0000 7f65ecaac700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65e80f5410 0x7f65e8108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:17.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.695+0000 7f65ecaac700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65e80f5410 0x7f65e8108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:41934/0 (socket says 172.21.15.123:41934) 2024-01-13T11:54:17.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.695+0000 7f65ecaac700 1 -- 172.21.15.123:0/2262800783 learned_addr learned my addr 172.21.15.123:0/2262800783 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:17.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.696+0000 7f65ecaac700 1 -- 172.21.15.123:0/2262800783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65dc005040 con 0x7f65e80f5410 2024-01-13T11:54:17.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.696+0000 7f65ecaac700 1 --2- 172.21.15.123:0/2262800783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65e80f5410 0x7f65e8108910 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f65dc013510 tx=0x7f65dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:17.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.696+0000 7f65e57fa700 1 -- 172.21.15.123:0/2262800783 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65dc009320 con 0x7f65e80f5410 2024-01-13T11:54:17.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.696+0000 7f65eed10700 1 -- 172.21.15.123:0/2262800783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e8109000 con 0x7f65e80f5410 2024-01-13T11:54:17.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.696+0000 7f65e57fa700 1 -- 172.21.15.123:0/2262800783 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f65dc013f00 con 0x7f65e80f5410 2024-01-13T11:54:17.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.696+0000 7f65e57fa700 1 -- 172.21.15.123:0/2262800783 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65dc014d90 con 0x7f65e80f5410 2024-01-13T11:54:17.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.696+0000 7f65eed10700 1 -- 172.21.15.123:0/2262800783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e8109490 con 0x7f65e80f5410 2024-01-13T11:54:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.697+0000 7f65e57fa700 1 -- 172.21.15.123:0/2262800783 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f65dc014420 con 0x7f65e80f5410 2024-01-13T11:54:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.697+0000 7f65eed10700 1 -- 172.21.15.123:0/2262800783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e8045ad0 con 0x7f65e80f5410 2024-01-13T11:54:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.697+0000 7f65e57fa700 1 --2- 172.21.15.123:0/2262800783 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f65d802ff60 0x7f65d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.697+0000 7f65e57fa700 1 -- 172.21.15.123:0/2262800783 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f65dc040fb0 con 0x7f65e80f5410 2024-01-13T11:54:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.697+0000 7f65e7fff700 1 --2- 172.21.15.123:0/2262800783 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f65d802ff60 0x7f65d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.698+0000 7f65e7fff700 1 --2- 172.21.15.123:0/2262800783 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f65d802ff60 0x7f65d8032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f65d40096a0 tx=0x7f65d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:17.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.702+0000 7f65e57fa700 1 -- 172.21.15.123:0/2262800783 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f65dc0095f0 con 0x7f65e80f5410 2024-01-13T11:54:17.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.910+0000 7f65eed10700 1 -- 172.21.15.123:0/2262800783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65e80fae60 con 0x7f65e80f5410 2024-01-13T11:54:17.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.911+0000 7f65e57fa700 1 -- 172.21.15.123:0/2262800783 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f65dc01db70 con 0x7f65e80f5410 2024-01-13T11:54:17.899 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:17.899 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:17.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.912+0000 7f65eed10700 1 -- 172.21.15.123:0/2262800783 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f65d802ff60 msgr2=0x7f65d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:17.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.912+0000 7f65eed10700 1 --2- 172.21.15.123:0/2262800783 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f65d802ff60 0x7f65d8032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f65d40096a0 tx=0x7f65d4006b40).stop 2024-01-13T11:54:17.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.912+0000 7f65eed10700 1 -- 172.21.15.123:0/2262800783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65e80f5410 msgr2=0x7f65e8108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:17.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.912+0000 7f65eed10700 1 --2- 172.21.15.123:0/2262800783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65e80f5410 0x7f65e8108910 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f65dc013510 tx=0x7f65dc00b040).stop 2024-01-13T11:54:17.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.912+0000 7f65eed10700 1 -- 172.21.15.123:0/2262800783 shutdown_connections 2024-01-13T11:54:17.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.912+0000 7f65eed10700 1 --2- 172.21.15.123:0/2262800783 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f65d802ff60 0x7f65d8032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:17.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.912+0000 7f65eed10700 1 --2- 172.21.15.123:0/2262800783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65e80f5410 0x7f65e8108910 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:17.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.912+0000 7f65eed10700 1 -- 172.21.15.123:0/2262800783 >> 172.21.15.123:0/2262800783 conn(0x7f65e80f0b70 msgr2=0x7f65e805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:17.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.912+0000 7f65eed10700 1 -- 172.21.15.123:0/2262800783 shutdown_connections 2024-01-13T11:54:17.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:17.912+0000 7f65eed10700 1 -- 172.21.15.123:0/2262800783 wait complete. 2024-01-13T11:54:17.901 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:18.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:18 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:17.911935+0000 mon.smithi059 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.123:0/2262800783' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:19.279 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:19.280 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:19.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:19 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:17.543897+0000 mgr.smithi059.hmyldj (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:20.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.425+0000 7efc08092700 1 -- 172.21.15.123:0/1666694967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc000f5410 msgr2=0x7efc000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:20.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.425+0000 7efc08092700 1 --2- 172.21.15.123:0/1666694967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc000f5410 0x7efc000f5830 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7efbf0009fb0 tx=0x7efbf0009b30).stop 2024-01-13T11:54:20.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.425+0000 7efc08092700 1 -- 172.21.15.123:0/1666694967 shutdown_connections 2024-01-13T11:54:20.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.425+0000 7efc08092700 1 --2- 172.21.15.123:0/1666694967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc000f5410 0x7efc000f5830 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:20.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.425+0000 7efc08092700 1 -- 172.21.15.123:0/1666694967 >> 172.21.15.123:0/1666694967 conn(0x7efc000f0b70 msgr2=0x7efc000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:20.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.426+0000 7efc08092700 1 -- 172.21.15.123:0/1666694967 shutdown_connections 2024-01-13T11:54:20.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.426+0000 7efc08092700 1 -- 172.21.15.123:0/1666694967 wait complete. 2024-01-13T11:54:20.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.426+0000 7efc08092700 1 Processor -- start 2024-01-13T11:54:20.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.426+0000 7efc08092700 1 -- start start 2024-01-13T11:54:20.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.426+0000 7efc08092700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc000f5410 0x7efc000fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:20.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.426+0000 7efc08092700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc000fb820 con 0x7efc000f5410 2024-01-13T11:54:20.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.426+0000 7efc05e2e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc000f5410 0x7efc000fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:20.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.426+0000 7efc05e2e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc000f5410 0x7efc000fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:41940/0 (socket says 172.21.15.123:41940) 2024-01-13T11:54:20.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.426+0000 7efc05e2e700 1 -- 172.21.15.123:0/63704305 learned_addr learned my addr 172.21.15.123:0/63704305 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:20.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.427+0000 7efc05e2e700 1 -- 172.21.15.123:0/63704305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf0005040 con 0x7efc000f5410 2024-01-13T11:54:20.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.427+0000 7efc05e2e700 1 --2- 172.21.15.123:0/63704305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc000f5410 0x7efc000fd3a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7efbf0014000 tx=0x7efbf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:20.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.427+0000 7efbf67fc700 1 -- 172.21.15.123:0/63704305 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbf00095f0 con 0x7efc000f5410 2024-01-13T11:54:20.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.427+0000 7efc08092700 1 -- 172.21.15.123:0/63704305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc000fb9c0 con 0x7efc000f5410 2024-01-13T11:54:20.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.427+0000 7efbf67fc700 1 -- 172.21.15.123:0/63704305 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efbf0007e70 con 0x7efc000f5410 2024-01-13T11:54:20.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.427+0000 7efbf67fc700 1 -- 172.21.15.123:0/63704305 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbf0014ea0 con 0x7efc000f5410 2024-01-13T11:54:20.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.427+0000 7efc08092700 1 -- 172.21.15.123:0/63704305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc000fbe50 con 0x7efc000f5410 2024-01-13T11:54:20.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.428+0000 7efbf67fc700 1 -- 172.21.15.123:0/63704305 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efbf00079c0 con 0x7efc000f5410 2024-01-13T11:54:20.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.428+0000 7efc08092700 1 -- 172.21.15.123:0/63704305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc00059100 con 0x7efc000f5410 2024-01-13T11:54:20.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.428+0000 7efbf67fc700 1 --2- 172.21.15.123:0/63704305 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7efbec02ffb0 0x7efbec032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:20.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.428+0000 7efbf67fc700 1 -- 172.21.15.123:0/63704305 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efbf00407f0 con 0x7efc000f5410 2024-01-13T11:54:20.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.428+0000 7efc0562d700 1 --2- 172.21.15.123:0/63704305 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7efbec02ffb0 0x7efbec032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:20.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.429+0000 7efc0562d700 1 --2- 172.21.15.123:0/63704305 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7efbec02ffb0 0x7efbec032470 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7efbfc0096a0 tx=0x7efbfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:20.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.433+0000 7efbf67fc700 1 -- 172.21.15.123:0/63704305 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efbf000f950 con 0x7efc000f5410 2024-01-13T11:54:20.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.640+0000 7efc08092700 1 -- 172.21.15.123:0/63704305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc000fae60 con 0x7efc000f5410 2024-01-13T11:54:20.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.640+0000 7efbf67fc700 1 -- 172.21.15.123:0/63704305 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efbf0010230 con 0x7efc000f5410 2024-01-13T11:54:20.629 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:20.629 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:20.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.642+0000 7efc08092700 1 -- 172.21.15.123:0/63704305 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7efbec02ffb0 msgr2=0x7efbec032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:20.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.642+0000 7efc08092700 1 --2- 172.21.15.123:0/63704305 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7efbec02ffb0 0x7efbec032470 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7efbfc0096a0 tx=0x7efbfc006b40).stop 2024-01-13T11:54:20.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.642+0000 7efc08092700 1 -- 172.21.15.123:0/63704305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc000f5410 msgr2=0x7efc000fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:20.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.642+0000 7efc08092700 1 --2- 172.21.15.123:0/63704305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc000f5410 0x7efc000fd3a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7efbf0014000 tx=0x7efbf000b040).stop 2024-01-13T11:54:20.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.642+0000 7efc08092700 1 -- 172.21.15.123:0/63704305 shutdown_connections 2024-01-13T11:54:20.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.642+0000 7efc08092700 1 --2- 172.21.15.123:0/63704305 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7efbec02ffb0 0x7efbec032470 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:20.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.642+0000 7efc08092700 1 --2- 172.21.15.123:0/63704305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc000f5410 0x7efc000fd3a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:20.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.642+0000 7efc08092700 1 -- 172.21.15.123:0/63704305 >> 172.21.15.123:0/63704305 conn(0x7efc000f0b70 msgr2=0x7efc0005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:20.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.642+0000 7efc08092700 1 -- 172.21.15.123:0/63704305 shutdown_connections 2024-01-13T11:54:20.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:20.642+0000 7efc08092700 1 -- 172.21.15.123:0/63704305 wait complete. 2024-01-13T11:54:20.631 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:21.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:21 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:19.544210+0000 mgr.smithi059.hmyldj (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:21.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:21 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:20.641674+0000 mon.smithi059 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.123:0/63704305' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:22.033 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:22.034 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:23.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.133+0000 7f2178eb1700 1 -- 172.21.15.123:0/1057939821 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21740f5410 msgr2=0x7f21740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:23.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.133+0000 7f2178eb1700 1 --2- 172.21.15.123:0/1057939821 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21740f5410 0x7f21740f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f216c009960 tx=0x7f216c004d10).stop 2024-01-13T11:54:23.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.133+0000 7f2178eb1700 1 -- 172.21.15.123:0/1057939821 shutdown_connections 2024-01-13T11:54:23.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.133+0000 7f2178eb1700 1 --2- 172.21.15.123:0/1057939821 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21740f5410 0x7f21740f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:23.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.133+0000 7f2178eb1700 1 -- 172.21.15.123:0/1057939821 >> 172.21.15.123:0/1057939821 conn(0x7f21740f0b70 msgr2=0x7f21740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:23.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.133+0000 7f2178eb1700 1 -- 172.21.15.123:0/1057939821 shutdown_connections 2024-01-13T11:54:23.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.133+0000 7f2178eb1700 1 -- 172.21.15.123:0/1057939821 wait complete. 2024-01-13T11:54:23.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.134+0000 7f2178eb1700 1 Processor -- start 2024-01-13T11:54:23.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.134+0000 7f2178eb1700 1 -- start start 2024-01-13T11:54:23.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.134+0000 7f2178eb1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21740f5410 0x7f2174106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:23.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.134+0000 7f2178eb1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2174106c20 con 0x7f21740f5410 2024-01-13T11:54:23.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.134+0000 7f217259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21740f5410 0x7f2174106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:23.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.134+0000 7f217259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21740f5410 0x7f2174106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:41954/0 (socket says 172.21.15.123:41954) 2024-01-13T11:54:23.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.134+0000 7f217259c700 1 -- 172.21.15.123:0/1570210562 learned_addr learned my addr 172.21.15.123:0/1570210562 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:23.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.135+0000 7f217259c700 1 -- 172.21.15.123:0/1570210562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216c009640 con 0x7f21740f5410 2024-01-13T11:54:23.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.135+0000 7f217259c700 1 --2- 172.21.15.123:0/1570210562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21740f5410 0x7f2174106730 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f216c004b70 tx=0x7f216c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:23.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.135+0000 7f2162ffd700 1 -- 172.21.15.123:0/1570210562 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f216c018070 con 0x7f21740f5410 2024-01-13T11:54:23.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.135+0000 7f2162ffd700 1 -- 172.21.15.123:0/1570210562 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f216c00b870 con 0x7f21740f5410 2024-01-13T11:54:23.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.135+0000 7f2178eb1700 1 -- 172.21.15.123:0/1570210562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2174106e20 con 0x7f21740f5410 2024-01-13T11:54:23.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.135+0000 7f2162ffd700 1 -- 172.21.15.123:0/1570210562 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f216c01c560 con 0x7f21740f5410 2024-01-13T11:54:23.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.135+0000 7f2178eb1700 1 -- 172.21.15.123:0/1570210562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21741072b0 con 0x7f21740f5410 2024-01-13T11:54:23.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.136+0000 7f2162ffd700 1 -- 172.21.15.123:0/1570210562 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f216c013630 con 0x7f21740f5410 2024-01-13T11:54:23.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.136+0000 7f2178eb1700 1 -- 172.21.15.123:0/1570210562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2174045ad0 con 0x7f21740f5410 2024-01-13T11:54:23.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.136+0000 7f2162ffd700 1 --2- 172.21.15.123:0/1570210562 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f215c02ff60 0x7f215c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:23.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.136+0000 7f2162ffd700 1 -- 172.21.15.123:0/1570210562 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f216c041250 con 0x7f21740f5410 2024-01-13T11:54:23.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.136+0000 7f2171d9b700 1 --2- 172.21.15.123:0/1570210562 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f215c02ff60 0x7f215c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:23.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.137+0000 7f2171d9b700 1 --2- 172.21.15.123:0/1570210562 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f215c02ff60 0x7f215c032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f21680096a0 tx=0x7f2168006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:23.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.141+0000 7f2162ffd700 1 -- 172.21.15.123:0/1570210562 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f216c004810 con 0x7f21740f5410 2024-01-13T11:54:23.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.349+0000 7f2178eb1700 1 -- 172.21.15.123:0/1570210562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21740fae60 con 0x7f21740f5410 2024-01-13T11:54:23.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.350+0000 7f2162ffd700 1 -- 172.21.15.123:0/1570210562 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f216c012380 con 0x7f21740f5410 2024-01-13T11:54:23.338 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:23.339 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:23.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.352+0000 7f2178eb1700 1 -- 172.21.15.123:0/1570210562 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f215c02ff60 msgr2=0x7f215c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:23.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.352+0000 7f2178eb1700 1 --2- 172.21.15.123:0/1570210562 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f215c02ff60 0x7f215c032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f21680096a0 tx=0x7f2168006b40).stop 2024-01-13T11:54:23.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.352+0000 7f2178eb1700 1 -- 172.21.15.123:0/1570210562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21740f5410 msgr2=0x7f2174106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:23.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.352+0000 7f2178eb1700 1 --2- 172.21.15.123:0/1570210562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21740f5410 0x7f2174106730 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f216c004b70 tx=0x7f216c004d10).stop 2024-01-13T11:54:23.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.352+0000 7f2178eb1700 1 -- 172.21.15.123:0/1570210562 shutdown_connections 2024-01-13T11:54:23.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.352+0000 7f2178eb1700 1 --2- 172.21.15.123:0/1570210562 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f215c02ff60 0x7f215c032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:23.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.352+0000 7f2178eb1700 1 --2- 172.21.15.123:0/1570210562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21740f5410 0x7f2174106730 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:23.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.352+0000 7f2178eb1700 1 -- 172.21.15.123:0/1570210562 >> 172.21.15.123:0/1570210562 conn(0x7f21740f0b70 msgr2=0x7f21740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:23.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.352+0000 7f2178eb1700 1 -- 172.21.15.123:0/1570210562 shutdown_connections 2024-01-13T11:54:23.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:23.352+0000 7f2178eb1700 1 -- 172.21.15.123:0/1570210562 wait complete. 2024-01-13T11:54:23.341 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:23.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:23 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:21.544675+0000 mgr.smithi059.hmyldj (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:24.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:24 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:23.351318+0000 mon.smithi059 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.123:0/1570210562' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:24.732 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:24.732 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:25.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:25 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:23.544992+0000 mgr.smithi059.hmyldj (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.917+0000 7faceead2700 1 -- 172.21.15.123:0/819835577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face80f6510 msgr2=0x7face80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.917+0000 7faceead2700 1 --2- 172.21.15.123:0/819835577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face80f6510 0x7face80f6930 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7facdc009fb0 tx=0x7facdc009b30).stop 2024-01-13T11:54:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.917+0000 7faceead2700 1 -- 172.21.15.123:0/819835577 shutdown_connections 2024-01-13T11:54:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.917+0000 7faceead2700 1 --2- 172.21.15.123:0/819835577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face80f6510 0x7face80f6930 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.917+0000 7faceead2700 1 -- 172.21.15.123:0/819835577 >> 172.21.15.123:0/819835577 conn(0x7face80f1c70 msgr2=0x7face80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.917+0000 7faceead2700 1 -- 172.21.15.123:0/819835577 shutdown_connections 2024-01-13T11:54:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.917+0000 7faceead2700 1 -- 172.21.15.123:0/819835577 wait complete. 2024-01-13T11:54:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.917+0000 7faceead2700 1 Processor -- start 2024-01-13T11:54:25.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.917+0000 7faceead2700 1 -- start start 2024-01-13T11:54:25.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.918+0000 7faceead2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face80f6510 0x7face8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:25.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.918+0000 7faceead2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face8106c20 con 0x7face80f6510 2024-01-13T11:54:25.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.918+0000 7facec86e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face80f6510 0x7face8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:25.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.918+0000 7facec86e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face80f6510 0x7face8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:41974/0 (socket says 172.21.15.123:41974) 2024-01-13T11:54:25.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.918+0000 7facec86e700 1 -- 172.21.15.123:0/1182040829 learned_addr learned my addr 172.21.15.123:0/1182040829 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:25.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.918+0000 7facec86e700 1 -- 172.21.15.123:0/1182040829 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc005040 con 0x7face80f6510 2024-01-13T11:54:25.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.918+0000 7facec86e700 1 --2- 172.21.15.123:0/1182040829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face80f6510 0x7face8106730 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7facdc013510 tx=0x7facdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:25.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.918+0000 7face57fa700 1 -- 172.21.15.123:0/1182040829 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facdc009320 con 0x7face80f6510 2024-01-13T11:54:25.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.918+0000 7face57fa700 1 -- 172.21.15.123:0/1182040829 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7facdc013f00 con 0x7face80f6510 2024-01-13T11:54:25.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.918+0000 7faceead2700 1 -- 172.21.15.123:0/1182040829 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face8106e20 con 0x7face80f6510 2024-01-13T11:54:25.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.918+0000 7face57fa700 1 -- 172.21.15.123:0/1182040829 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facdc014d90 con 0x7face80f6510 2024-01-13T11:54:25.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.918+0000 7faceead2700 1 -- 172.21.15.123:0/1182040829 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face81072b0 con 0x7face80f6510 2024-01-13T11:54:25.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.919+0000 7face57fa700 1 -- 172.21.15.123:0/1182040829 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7facdc014420 con 0x7face80f6510 2024-01-13T11:54:25.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.919+0000 7face57fa700 1 --2- 172.21.15.123:0/1182040829 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7facd802ff60 0x7facd8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:25.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.919+0000 7face57fa700 1 -- 172.21.15.123:0/1182040829 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7facdc041070 con 0x7face80f6510 2024-01-13T11:54:25.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.919+0000 7faceead2700 1 -- 172.21.15.123:0/1182040829 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face8045ad0 con 0x7face80f6510 2024-01-13T11:54:25.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.920+0000 7face7fff700 1 --2- 172.21.15.123:0/1182040829 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7facd802ff60 0x7facd8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:25.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.920+0000 7face7fff700 1 --2- 172.21.15.123:0/1182040829 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7facd802ff60 0x7facd8032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7facd40096a0 tx=0x7facd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:25.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:25.924+0000 7face57fa700 1 -- 172.21.15.123:0/1182040829 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7facdc0095f0 con 0x7face80f6510 2024-01-13T11:54:26.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:26.133+0000 7faceead2700 1 -- 172.21.15.123:0/1182040829 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7face80fc050 con 0x7face80f6510 2024-01-13T11:54:26.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:26.134+0000 7face57fa700 1 -- 172.21.15.123:0/1182040829 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7facdc01db70 con 0x7face80f6510 2024-01-13T11:54:26.122 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:26.123 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:26.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:26.135+0000 7faceead2700 1 -- 172.21.15.123:0/1182040829 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7facd802ff60 msgr2=0x7facd8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:26.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:26.135+0000 7faceead2700 1 --2- 172.21.15.123:0/1182040829 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7facd802ff60 0x7facd8032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7facd40096a0 tx=0x7facd4006b40).stop 2024-01-13T11:54:26.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:26.135+0000 7faceead2700 1 -- 172.21.15.123:0/1182040829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face80f6510 msgr2=0x7face8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:26.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:26.135+0000 7faceead2700 1 --2- 172.21.15.123:0/1182040829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face80f6510 0x7face8106730 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7facdc013510 tx=0x7facdc00b040).stop 2024-01-13T11:54:26.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:26.136+0000 7faceead2700 1 -- 172.21.15.123:0/1182040829 shutdown_connections 2024-01-13T11:54:26.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:26.136+0000 7faceead2700 1 --2- 172.21.15.123:0/1182040829 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7facd802ff60 0x7facd8032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:26.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:26.136+0000 7faceead2700 1 --2- 172.21.15.123:0/1182040829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7face80f6510 0x7face8106730 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:26.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:26.136+0000 7faceead2700 1 -- 172.21.15.123:0/1182040829 >> 172.21.15.123:0/1182040829 conn(0x7face80f1c70 msgr2=0x7face80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:26.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:26.136+0000 7faceead2700 1 -- 172.21.15.123:0/1182040829 shutdown_connections 2024-01-13T11:54:26.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:26.136+0000 7faceead2700 1 -- 172.21.15.123:0/1182040829 wait complete. 2024-01-13T11:54:26.125 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:26.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:26 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:26.135289+0000 mon.smithi059 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.123:0/1182040829' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:27.504 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:27.504 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:27.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:27 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:25.545293+0000 mgr.smithi059.hmyldj (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:28.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.666+0000 7f8579235700 1 -- 172.21.15.123:0/1048304638 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85740f5410 msgr2=0x7f85740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:28.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.666+0000 7f8579235700 1 --2- 172.21.15.123:0/1048304638 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85740f5410 0x7f85740f5830 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f8568009fb0 tx=0x7f8568009b30).stop 2024-01-13T11:54:28.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.666+0000 7f8579235700 1 -- 172.21.15.123:0/1048304638 shutdown_connections 2024-01-13T11:54:28.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.666+0000 7f8579235700 1 --2- 172.21.15.123:0/1048304638 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85740f5410 0x7f85740f5830 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:28.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.666+0000 7f8579235700 1 -- 172.21.15.123:0/1048304638 >> 172.21.15.123:0/1048304638 conn(0x7f85740f0b70 msgr2=0x7f85740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:28.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.666+0000 7f8579235700 1 -- 172.21.15.123:0/1048304638 shutdown_connections 2024-01-13T11:54:28.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.666+0000 7f8579235700 1 -- 172.21.15.123:0/1048304638 wait complete. 2024-01-13T11:54:28.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.667+0000 7f8579235700 1 Processor -- start 2024-01-13T11:54:28.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.667+0000 7f8579235700 1 -- start start 2024-01-13T11:54:28.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.667+0000 7f8579235700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8574105740 0x7f8574105b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:28.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.667+0000 7f8579235700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8574106050 con 0x7f8574105740 2024-01-13T11:54:28.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.667+0000 7f8572d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8574105740 0x7f8574105b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:28.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.667+0000 7f8572d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8574105740 0x7f8574105b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:60122/0 (socket says 172.21.15.123:60122) 2024-01-13T11:54:28.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.667+0000 7f8572d9d700 1 -- 172.21.15.123:0/522540828 learned_addr learned my addr 172.21.15.123:0/522540828 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:28.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.667+0000 7f8572d9d700 1 -- 172.21.15.123:0/522540828 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8568005040 con 0x7f8574105740 2024-01-13T11:54:28.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.667+0000 7f8572d9d700 1 --2- 172.21.15.123:0/522540828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8574105740 0x7f8574105b60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f8568009f80 tx=0x7f8568009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:28.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.667+0000 7f855b7fe700 1 -- 172.21.15.123:0/522540828 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85680136c0 con 0x7f8574105740 2024-01-13T11:54:28.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.667+0000 7f8579235700 1 -- 172.21.15.123:0/522540828 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8574106250 con 0x7f8574105740 2024-01-13T11:54:28.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.668+0000 7f855b7fe700 1 -- 172.21.15.123:0/522540828 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f85680095f0 con 0x7f8574105740 2024-01-13T11:54:28.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.668+0000 7f855b7fe700 1 -- 172.21.15.123:0/522540828 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8568014420 con 0x7f8574105740 2024-01-13T11:54:28.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.668+0000 7f8579235700 1 -- 172.21.15.123:0/522540828 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8574106da0 con 0x7f8574105740 2024-01-13T11:54:28.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.668+0000 7f855b7fe700 1 -- 172.21.15.123:0/522540828 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8568025060 con 0x7f8574105740 2024-01-13T11:54:28.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.668+0000 7f8579235700 1 -- 172.21.15.123:0/522540828 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8574059100 con 0x7f8574105740 2024-01-13T11:54:28.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.668+0000 7f855b7fe700 1 --2- 172.21.15.123:0/522540828 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f855c02ff60 0x7f855c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:28.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.669+0000 7f855b7fe700 1 -- 172.21.15.123:0/522540828 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8568040d80 con 0x7f8574105740 2024-01-13T11:54:28.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.669+0000 7f857259c700 1 --2- 172.21.15.123:0/522540828 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f855c02ff60 0x7f855c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:28.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.669+0000 7f857259c700 1 --2- 172.21.15.123:0/522540828 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f855c02ff60 0x7f855c032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f85640096a0 tx=0x7f8564006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:28.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.674+0000 7f855b7fe700 1 -- 172.21.15.123:0/522540828 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f856801cb60 con 0x7f8574105740 2024-01-13T11:54:28.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.879+0000 7f8579235700 1 -- 172.21.15.123:0/522540828 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8574107560 con 0x7f8574105740 2024-01-13T11:54:28.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.880+0000 7f855b7fe700 1 -- 172.21.15.123:0/522540828 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8568042070 con 0x7f8574105740 2024-01-13T11:54:28.868 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:28.868 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:28.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.881+0000 7f8579235700 1 -- 172.21.15.123:0/522540828 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f855c02ff60 msgr2=0x7f855c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:28.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.881+0000 7f8579235700 1 --2- 172.21.15.123:0/522540828 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f855c02ff60 0x7f855c032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f85640096a0 tx=0x7f8564006b40).stop 2024-01-13T11:54:28.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.881+0000 7f8579235700 1 -- 172.21.15.123:0/522540828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8574105740 msgr2=0x7f8574105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:28.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.881+0000 7f8579235700 1 --2- 172.21.15.123:0/522540828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8574105740 0x7f8574105b60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f8568009f80 tx=0x7f8568009b30).stop 2024-01-13T11:54:28.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.881+0000 7f8579235700 1 -- 172.21.15.123:0/522540828 shutdown_connections 2024-01-13T11:54:28.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.881+0000 7f8579235700 1 --2- 172.21.15.123:0/522540828 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f855c02ff60 0x7f855c032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:28.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.881+0000 7f8579235700 1 --2- 172.21.15.123:0/522540828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8574105740 0x7f8574105b60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:28.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.881+0000 7f8579235700 1 -- 172.21.15.123:0/522540828 >> 172.21.15.123:0/522540828 conn(0x7f85740f0b70 msgr2=0x7f85740f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:28.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.881+0000 7f8579235700 1 -- 172.21.15.123:0/522540828 shutdown_connections 2024-01-13T11:54:28.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:28.881+0000 7f8579235700 1 -- 172.21.15.123:0/522540828 wait complete. 2024-01-13T11:54:28.871 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:29.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:29 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:27.545716+0000 mgr.smithi059.hmyldj (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:29.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:29 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:28.881152+0000 mon.smithi059 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.123:0/522540828' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:30.259 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:30.259 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:31.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.391+0000 7ffbbaf89700 1 -- 172.21.15.123:0/1841224699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbb40f5410 msgr2=0x7ffbb40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:31.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.391+0000 7ffbbaf89700 1 --2- 172.21.15.123:0/1841224699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbb40f5410 0x7ffbb40f5830 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7ffba8009fb0 tx=0x7ffba8009b30).stop 2024-01-13T11:54:31.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.391+0000 7ffbbaf89700 1 -- 172.21.15.123:0/1841224699 shutdown_connections 2024-01-13T11:54:31.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.391+0000 7ffbbaf89700 1 --2- 172.21.15.123:0/1841224699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbb40f5410 0x7ffbb40f5830 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:31.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.391+0000 7ffbbaf89700 1 -- 172.21.15.123:0/1841224699 >> 172.21.15.123:0/1841224699 conn(0x7ffbb40f0b70 msgr2=0x7ffbb40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:31.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.391+0000 7ffbbaf89700 1 -- 172.21.15.123:0/1841224699 shutdown_connections 2024-01-13T11:54:31.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.391+0000 7ffbbaf89700 1 -- 172.21.15.123:0/1841224699 wait complete. 2024-01-13T11:54:31.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.392+0000 7ffbbaf89700 1 Processor -- start 2024-01-13T11:54:31.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.392+0000 7ffbbaf89700 1 -- start start 2024-01-13T11:54:31.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.392+0000 7ffbbaf89700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbb40f5410 0x7ffbb4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:31.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.392+0000 7ffbbaf89700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb4105b20 con 0x7ffbb40f5410 2024-01-13T11:54:31.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.392+0000 7ffbb8d25700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbb40f5410 0x7ffbb4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:31.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.392+0000 7ffbb8d25700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbb40f5410 0x7ffbb4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:60136/0 (socket says 172.21.15.123:60136) 2024-01-13T11:54:31.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.392+0000 7ffbb8d25700 1 -- 172.21.15.123:0/3456781950 learned_addr learned my addr 172.21.15.123:0/3456781950 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:31.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.393+0000 7ffbb8d25700 1 -- 172.21.15.123:0/3456781950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba8005040 con 0x7ffbb40f5410 2024-01-13T11:54:31.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.393+0000 7ffbb8d25700 1 --2- 172.21.15.123:0/3456781950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbb40f5410 0x7ffbb4105630 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7ffba8013510 tx=0x7ffba800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:31.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.393+0000 7ffbb17fa700 1 -- 172.21.15.123:0/3456781950 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffba8009320 con 0x7ffbb40f5410 2024-01-13T11:54:31.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.393+0000 7ffbbaf89700 1 -- 172.21.15.123:0/3456781950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb4105d20 con 0x7ffbb40f5410 2024-01-13T11:54:31.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.393+0000 7ffbb17fa700 1 -- 172.21.15.123:0/3456781950 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffba8013f00 con 0x7ffbb40f5410 2024-01-13T11:54:31.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.393+0000 7ffbb17fa700 1 -- 172.21.15.123:0/3456781950 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffba8014d90 con 0x7ffbb40f5410 2024-01-13T11:54:31.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.393+0000 7ffbbaf89700 1 -- 172.21.15.123:0/3456781950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb41061b0 con 0x7ffbb40f5410 2024-01-13T11:54:31.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.394+0000 7ffbb17fa700 1 -- 172.21.15.123:0/3456781950 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffba80263a0 con 0x7ffbb40f5410 2024-01-13T11:54:31.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.394+0000 7ffbb17fa700 1 --2- 172.21.15.123:0/3456781950 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffba402ff60 0x7ffba4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:31.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.394+0000 7ffbb17fa700 1 -- 172.21.15.123:0/3456781950 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffba80410a0 con 0x7ffbb40f5410 2024-01-13T11:54:31.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.394+0000 7ffbbaf89700 1 -- 172.21.15.123:0/3456781950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb4045ad0 con 0x7ffbb40f5410 2024-01-13T11:54:31.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.394+0000 7ffbb3fff700 1 --2- 172.21.15.123:0/3456781950 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffba402ff60 0x7ffba4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:31.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.395+0000 7ffbb3fff700 1 --2- 172.21.15.123:0/3456781950 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffba402ff60 0x7ffba4032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7ffba00096a0 tx=0x7ffba0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:31.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.399+0000 7ffbb17fa700 1 -- 172.21.15.123:0/3456781950 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffba8012920 con 0x7ffbb40f5410 2024-01-13T11:54:31.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.606+0000 7ffbbaf89700 1 -- 172.21.15.123:0/3456781950 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffbb40faf50 con 0x7ffbb40f5410 2024-01-13T11:54:31.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.607+0000 7ffbb17fa700 1 -- 172.21.15.123:0/3456781950 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffba802b020 con 0x7ffbb40f5410 2024-01-13T11:54:31.595 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:31.595 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:31.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.608+0000 7ffbbaf89700 1 -- 172.21.15.123:0/3456781950 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffba402ff60 msgr2=0x7ffba4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:31.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.608+0000 7ffbbaf89700 1 --2- 172.21.15.123:0/3456781950 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffba402ff60 0x7ffba4032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7ffba00096a0 tx=0x7ffba0006b40).stop 2024-01-13T11:54:31.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.608+0000 7ffbbaf89700 1 -- 172.21.15.123:0/3456781950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbb40f5410 msgr2=0x7ffbb4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:31.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.608+0000 7ffbbaf89700 1 --2- 172.21.15.123:0/3456781950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbb40f5410 0x7ffbb4105630 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7ffba8013510 tx=0x7ffba800b040).stop 2024-01-13T11:54:31.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.608+0000 7ffbbaf89700 1 -- 172.21.15.123:0/3456781950 shutdown_connections 2024-01-13T11:54:31.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.608+0000 7ffbbaf89700 1 --2- 172.21.15.123:0/3456781950 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffba402ff60 0x7ffba4032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:31.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.608+0000 7ffbbaf89700 1 --2- 172.21.15.123:0/3456781950 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbb40f5410 0x7ffbb4105630 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:31.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.608+0000 7ffbbaf89700 1 -- 172.21.15.123:0/3456781950 >> 172.21.15.123:0/3456781950 conn(0x7ffbb40f0b70 msgr2=0x7ffbb40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:31.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.608+0000 7ffbbaf89700 1 -- 172.21.15.123:0/3456781950 shutdown_connections 2024-01-13T11:54:31.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:31.608+0000 7ffbbaf89700 1 -- 172.21.15.123:0/3456781950 wait complete. 2024-01-13T11:54:31.600 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:31.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:31 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:29.546047+0000 mgr.smithi059.hmyldj (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:32.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:32 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:31.607787+0000 mon.smithi059 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.123:0/3456781950' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:32.991 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:32.991 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:33.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:33 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:31.546366+0000 mgr.smithi059.hmyldj (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:34.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.117+0000 7f186d698700 1 -- 172.21.15.123:0/1073221314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18680f51a0 msgr2=0x7f18680f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:34.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.117+0000 7f186d698700 1 --2- 172.21.15.123:0/1073221314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18680f51a0 0x7f18680f55c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f1858009fb0 tx=0x7f1858009b30).stop 2024-01-13T11:54:34.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.118+0000 7f186d698700 1 -- 172.21.15.123:0/1073221314 shutdown_connections 2024-01-13T11:54:34.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.118+0000 7f186d698700 1 --2- 172.21.15.123:0/1073221314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18680f51a0 0x7f18680f55c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:34.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.118+0000 7f186d698700 1 -- 172.21.15.123:0/1073221314 >> 172.21.15.123:0/1073221314 conn(0x7f18680f09a0 msgr2=0x7f18680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:34.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.118+0000 7f186d698700 1 -- 172.21.15.123:0/1073221314 shutdown_connections 2024-01-13T11:54:34.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.118+0000 7f186d698700 1 -- 172.21.15.123:0/1073221314 wait complete. 2024-01-13T11:54:34.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.118+0000 7f186d698700 1 Processor -- start 2024-01-13T11:54:34.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.118+0000 7f186d698700 1 -- start start 2024-01-13T11:54:34.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.118+0000 7f186d698700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1868106610 0x7f1868106a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:34.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.118+0000 7f186d698700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1868106f20 con 0x7f1868106610 2024-01-13T11:54:34.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.119+0000 7f1867fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1868106610 0x7f1868106a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:34.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.119+0000 7f1867fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1868106610 0x7f1868106a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:60154/0 (socket says 172.21.15.123:60154) 2024-01-13T11:54:34.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.119+0000 7f1867fff700 1 -- 172.21.15.123:0/2567303191 learned_addr learned my addr 172.21.15.123:0/2567303191 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:34.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.119+0000 7f1867fff700 1 -- 172.21.15.123:0/2567303191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1858005040 con 0x7f1868106610 2024-01-13T11:54:34.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.119+0000 7f1867fff700 1 --2- 172.21.15.123:0/2567303191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1868106610 0x7f1868106a30 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f1858009f80 tx=0x7f1858009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:34.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.119+0000 7f1864ff9700 1 -- 172.21.15.123:0/2567303191 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18580136c0 con 0x7f1868106610 2024-01-13T11:54:34.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.119+0000 7f186d698700 1 -- 172.21.15.123:0/2567303191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1868107120 con 0x7f1868106610 2024-01-13T11:54:34.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.119+0000 7f1864ff9700 1 -- 172.21.15.123:0/2567303191 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f18580095f0 con 0x7f1868106610 2024-01-13T11:54:34.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.119+0000 7f1864ff9700 1 -- 172.21.15.123:0/2567303191 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1858014420 con 0x7f1868106610 2024-01-13T11:54:34.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.119+0000 7f186d698700 1 -- 172.21.15.123:0/2567303191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1868107c70 con 0x7f1868106610 2024-01-13T11:54:34.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.120+0000 7f1864ff9700 1 -- 172.21.15.123:0/2567303191 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1858025060 con 0x7f1868106610 2024-01-13T11:54:34.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.120+0000 7f1864ff9700 1 --2- 172.21.15.123:0/2567303191 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f185002ff60 0x7f1850032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:34.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.120+0000 7f1864ff9700 1 -- 172.21.15.123:0/2567303191 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1858040eb0 con 0x7f1868106610 2024-01-13T11:54:34.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.120+0000 7f186d698700 1 -- 172.21.15.123:0/2567303191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1868059100 con 0x7f1868106610 2024-01-13T11:54:34.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.121+0000 7f18677fe700 1 --2- 172.21.15.123:0/2567303191 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f185002ff60 0x7f1850032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:34.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.121+0000 7f18677fe700 1 --2- 172.21.15.123:0/2567303191 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f185002ff60 0x7f1850032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f185c0096a0 tx=0x7f185c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:34.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.125+0000 7f1864ff9700 1 -- 172.21.15.123:0/2567303191 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1858028440 con 0x7f1868106610 2024-01-13T11:54:34.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.333+0000 7f186d698700 1 -- 172.21.15.123:0/2567303191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1868045ad0 con 0x7f1868106610 2024-01-13T11:54:34.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.333+0000 7f1864ff9700 1 -- 172.21.15.123:0/2567303191 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1858004490 con 0x7f1868106610 2024-01-13T11:54:34.322 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:34.322 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:34.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.335+0000 7f186d698700 1 -- 172.21.15.123:0/2567303191 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f185002ff60 msgr2=0x7f1850032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.335+0000 7f186d698700 1 --2- 172.21.15.123:0/2567303191 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f185002ff60 0x7f1850032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f185c0096a0 tx=0x7f185c006b40).stop 2024-01-13T11:54:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.335+0000 7f186d698700 1 -- 172.21.15.123:0/2567303191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1868106610 msgr2=0x7f1868106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.335+0000 7f186d698700 1 --2- 172.21.15.123:0/2567303191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1868106610 0x7f1868106a30 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f1858009f80 tx=0x7f1858009b30).stop 2024-01-13T11:54:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.335+0000 7f186d698700 1 -- 172.21.15.123:0/2567303191 shutdown_connections 2024-01-13T11:54:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.335+0000 7f186d698700 1 --2- 172.21.15.123:0/2567303191 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f185002ff60 0x7f1850032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.335+0000 7f186d698700 1 --2- 172.21.15.123:0/2567303191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1868106610 0x7f1868106a30 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.335+0000 7f186d698700 1 -- 172.21.15.123:0/2567303191 >> 172.21.15.123:0/2567303191 conn(0x7f18680f09a0 msgr2=0x7f186805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.335+0000 7f186d698700 1 -- 172.21.15.123:0/2567303191 shutdown_connections 2024-01-13T11:54:34.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:34.335+0000 7f186d698700 1 -- 172.21.15.123:0/2567303191 wait complete. 2024-01-13T11:54:34.325 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:35.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:35 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:33.546652+0000 mgr.smithi059.hmyldj (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:35.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:35 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:34.334815+0000 mon.smithi059 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.123:0/2567303191' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:35.702 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:35.703 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:36.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.799+0000 7f5f0fbf9700 1 -- 172.21.15.123:0/4131296885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f080f6510 msgr2=0x7f5f080f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:36.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.799+0000 7f5f0fbf9700 1 --2- 172.21.15.123:0/4131296885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f080f6510 0x7f5f080f6930 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f5efc009fb0 tx=0x7f5efc009b30).stop 2024-01-13T11:54:36.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.799+0000 7f5f0fbf9700 1 -- 172.21.15.123:0/4131296885 shutdown_connections 2024-01-13T11:54:36.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.799+0000 7f5f0fbf9700 1 --2- 172.21.15.123:0/4131296885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f080f6510 0x7f5f080f6930 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:36.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.799+0000 7f5f0fbf9700 1 -- 172.21.15.123:0/4131296885 >> 172.21.15.123:0/4131296885 conn(0x7f5f080f1c70 msgr2=0x7f5f080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:36.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.799+0000 7f5f0fbf9700 1 -- 172.21.15.123:0/4131296885 shutdown_connections 2024-01-13T11:54:36.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.799+0000 7f5f0fbf9700 1 -- 172.21.15.123:0/4131296885 wait complete. 2024-01-13T11:54:36.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.800+0000 7f5f0fbf9700 1 Processor -- start 2024-01-13T11:54:36.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.800+0000 7f5f0fbf9700 1 -- start start 2024-01-13T11:54:36.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.800+0000 7f5f0fbf9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f080f6510 0x7f5f08108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:36.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.800+0000 7f5f0fbf9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f08108e00 con 0x7f5f080f6510 2024-01-13T11:54:36.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.800+0000 7f5f0d995700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f080f6510 0x7f5f08108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:36.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.800+0000 7f5f0d995700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f080f6510 0x7f5f08108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:44638/0 (socket says 172.21.15.123:44638) 2024-01-13T11:54:36.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.800+0000 7f5f0d995700 1 -- 172.21.15.123:0/3450890631 learned_addr learned my addr 172.21.15.123:0/3450890631 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:36.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.801+0000 7f5f0d995700 1 -- 172.21.15.123:0/3450890631 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5efc005040 con 0x7f5f080f6510 2024-01-13T11:54:36.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.801+0000 7f5f0d995700 1 --2- 172.21.15.123:0/3450890631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f080f6510 0x7f5f08108910 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f5efc013510 tx=0x7f5efc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:36.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.801+0000 7f5efa7fc700 1 -- 172.21.15.123:0/3450890631 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5efc009320 con 0x7f5f080f6510 2024-01-13T11:54:36.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.801+0000 7f5f0fbf9700 1 -- 172.21.15.123:0/3450890631 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f08109000 con 0x7f5f080f6510 2024-01-13T11:54:36.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.801+0000 7f5efa7fc700 1 -- 172.21.15.123:0/3450890631 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5efc013f00 con 0x7f5f080f6510 2024-01-13T11:54:36.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.801+0000 7f5efa7fc700 1 -- 172.21.15.123:0/3450890631 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5efc014d90 con 0x7f5f080f6510 2024-01-13T11:54:36.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.801+0000 7f5f0fbf9700 1 -- 172.21.15.123:0/3450890631 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f08109490 con 0x7f5f080f6510 2024-01-13T11:54:36.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.802+0000 7f5efa7fc700 1 -- 172.21.15.123:0/3450890631 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5efc0095f0 con 0x7f5f080f6510 2024-01-13T11:54:36.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.802+0000 7f5f0fbf9700 1 -- 172.21.15.123:0/3450890631 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f08045ad0 con 0x7f5f080f6510 2024-01-13T11:54:36.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.802+0000 7f5efa7fc700 1 --2- 172.21.15.123:0/3450890631 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5ef402ff60 0x7f5ef4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:36.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.802+0000 7f5efa7fc700 1 -- 172.21.15.123:0/3450890631 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5efc041080 con 0x7f5f080f6510 2024-01-13T11:54:36.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.802+0000 7f5f0d194700 1 --2- 172.21.15.123:0/3450890631 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5ef402ff60 0x7f5ef4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:36.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.803+0000 7f5f0d194700 1 --2- 172.21.15.123:0/3450890631 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5ef402ff60 0x7f5ef4032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f5f040096a0 tx=0x7f5f04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:36.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:36.807+0000 7f5efa7fc700 1 -- 172.21.15.123:0/3450890631 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5efc014420 con 0x7f5f080f6510 2024-01-13T11:54:37.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:37.011+0000 7f5f0fbf9700 1 -- 172.21.15.123:0/3450890631 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f0810a340 con 0x7f5f080f6510 2024-01-13T11:54:37.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:37.012+0000 7f5efa7fc700 1 -- 172.21.15.123:0/3450890631 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5efc010230 con 0x7f5f080f6510 2024-01-13T11:54:37.001 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:37.001 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:37.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:37.014+0000 7f5f0fbf9700 1 -- 172.21.15.123:0/3450890631 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5ef402ff60 msgr2=0x7f5ef4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:37.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:37.014+0000 7f5f0fbf9700 1 --2- 172.21.15.123:0/3450890631 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5ef402ff60 0x7f5ef4032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f5f040096a0 tx=0x7f5f04006b40).stop 2024-01-13T11:54:37.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:37.014+0000 7f5f0fbf9700 1 -- 172.21.15.123:0/3450890631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f080f6510 msgr2=0x7f5f08108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:37.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:37.014+0000 7f5f0fbf9700 1 --2- 172.21.15.123:0/3450890631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f080f6510 0x7f5f08108910 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f5efc013510 tx=0x7f5efc00b040).stop 2024-01-13T11:54:37.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:37.014+0000 7f5f0fbf9700 1 -- 172.21.15.123:0/3450890631 shutdown_connections 2024-01-13T11:54:37.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:37.014+0000 7f5f0fbf9700 1 --2- 172.21.15.123:0/3450890631 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5ef402ff60 0x7f5ef4032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:37.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:37.014+0000 7f5f0fbf9700 1 --2- 172.21.15.123:0/3450890631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f080f6510 0x7f5f08108910 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:37.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:37.014+0000 7f5f0fbf9700 1 -- 172.21.15.123:0/3450890631 >> 172.21.15.123:0/3450890631 conn(0x7f5f080f1c70 msgr2=0x7f5f080f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:37.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:37.014+0000 7f5f0fbf9700 1 -- 172.21.15.123:0/3450890631 shutdown_connections 2024-01-13T11:54:37.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:37.014+0000 7f5f0fbf9700 1 -- 172.21.15.123:0/3450890631 wait complete. 2024-01-13T11:54:37.003 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:37.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:37 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:35.547022+0000 mgr.smithi059.hmyldj (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:37.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:37 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:37.013540+0000 mon.smithi059 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.123:0/3450890631' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:38.386 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:38.386 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:39.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.556+0000 7f9ab64aa700 1 -- 172.21.15.123:0/4273621146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ab00f5410 msgr2=0x7f9ab00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:39.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.556+0000 7f9ab64aa700 1 --2- 172.21.15.123:0/4273621146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ab00f5410 0x7f9ab00f5830 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f9aa4009fb0 tx=0x7f9aa4009b30).stop 2024-01-13T11:54:39.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.556+0000 7f9ab64aa700 1 -- 172.21.15.123:0/4273621146 shutdown_connections 2024-01-13T11:54:39.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.556+0000 7f9ab64aa700 1 --2- 172.21.15.123:0/4273621146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ab00f5410 0x7f9ab00f5830 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:39.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.556+0000 7f9ab64aa700 1 -- 172.21.15.123:0/4273621146 >> 172.21.15.123:0/4273621146 conn(0x7f9ab00f0b70 msgr2=0x7f9ab00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:39.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.556+0000 7f9ab64aa700 1 -- 172.21.15.123:0/4273621146 shutdown_connections 2024-01-13T11:54:39.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.556+0000 7f9ab64aa700 1 -- 172.21.15.123:0/4273621146 wait complete. 2024-01-13T11:54:39.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.556+0000 7f9ab64aa700 1 Processor -- start 2024-01-13T11:54:39.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.557+0000 7f9ab64aa700 1 -- start start 2024-01-13T11:54:39.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.557+0000 7f9ab64aa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ab00f5410 0x7f9ab0106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:39.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.557+0000 7f9ab64aa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab0106c00 con 0x7f9ab00f5410 2024-01-13T11:54:39.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.557+0000 7f9aaffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ab00f5410 0x7f9ab0106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:39.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.557+0000 7f9aaffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ab00f5410 0x7f9ab0106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:44650/0 (socket says 172.21.15.123:44650) 2024-01-13T11:54:39.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.557+0000 7f9aaffff700 1 -- 172.21.15.123:0/1398395379 learned_addr learned my addr 172.21.15.123:0/1398395379 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:39.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.557+0000 7f9aaffff700 1 -- 172.21.15.123:0/1398395379 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa4005040 con 0x7f9ab00f5410 2024-01-13T11:54:39.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.557+0000 7f9aaffff700 1 --2- 172.21.15.123:0/1398395379 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ab00f5410 0x7f9ab0106710 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f9aa4013510 tx=0x7f9aa400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:39.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.558+0000 7f9aacff9700 1 -- 172.21.15.123:0/1398395379 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9aa4009320 con 0x7f9ab00f5410 2024-01-13T11:54:39.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.558+0000 7f9ab64aa700 1 -- 172.21.15.123:0/1398395379 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab0106e00 con 0x7f9ab00f5410 2024-01-13T11:54:39.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.558+0000 7f9aacff9700 1 -- 172.21.15.123:0/1398395379 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9aa4013f00 con 0x7f9ab00f5410 2024-01-13T11:54:39.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.558+0000 7f9aacff9700 1 -- 172.21.15.123:0/1398395379 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9aa4014d90 con 0x7f9ab00f5410 2024-01-13T11:54:39.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.558+0000 7f9ab64aa700 1 -- 172.21.15.123:0/1398395379 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab0107290 con 0x7f9ab00f5410 2024-01-13T11:54:39.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.558+0000 7f9aacff9700 1 -- 172.21.15.123:0/1398395379 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9aa4014420 con 0x7f9ab00f5410 2024-01-13T11:54:39.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.559+0000 7f9ab64aa700 1 -- 172.21.15.123:0/1398395379 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab0045ad0 con 0x7f9ab00f5410 2024-01-13T11:54:39.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.559+0000 7f9aacff9700 1 --2- 172.21.15.123:0/1398395379 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9a9802ff60 0x7f9a98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:39.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.559+0000 7f9aacff9700 1 -- 172.21.15.123:0/1398395379 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9aa4041020 con 0x7f9ab00f5410 2024-01-13T11:54:39.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.559+0000 7f9aaf7fe700 1 --2- 172.21.15.123:0/1398395379 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9a9802ff60 0x7f9a98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:39.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.559+0000 7f9aaf7fe700 1 --2- 172.21.15.123:0/1398395379 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9a9802ff60 0x7f9a98032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9aa00096a0 tx=0x7f9aa0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:39.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.564+0000 7f9aacff9700 1 -- 172.21.15.123:0/1398395379 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9aa40095f0 con 0x7f9ab00f5410 2024-01-13T11:54:39.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:39 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:37.547303+0000 mgr.smithi059.hmyldj (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:39.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.772+0000 7f9ab64aa700 1 -- 172.21.15.123:0/1398395379 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ab01080e0 con 0x7f9ab00f5410 2024-01-13T11:54:39.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.773+0000 7f9aacff9700 1 -- 172.21.15.123:0/1398395379 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9aa401db70 con 0x7f9ab00f5410 2024-01-13T11:54:39.760 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:39.760 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:39.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.774+0000 7f9ab64aa700 1 -- 172.21.15.123:0/1398395379 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9a9802ff60 msgr2=0x7f9a98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:39.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.774+0000 7f9ab64aa700 1 --2- 172.21.15.123:0/1398395379 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9a9802ff60 0x7f9a98032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9aa00096a0 tx=0x7f9aa0006b40).stop 2024-01-13T11:54:39.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.774+0000 7f9ab64aa700 1 -- 172.21.15.123:0/1398395379 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ab00f5410 msgr2=0x7f9ab0106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:39.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.774+0000 7f9ab64aa700 1 --2- 172.21.15.123:0/1398395379 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ab00f5410 0x7f9ab0106710 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f9aa4013510 tx=0x7f9aa400b040).stop 2024-01-13T11:54:39.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.774+0000 7f9ab64aa700 1 -- 172.21.15.123:0/1398395379 shutdown_connections 2024-01-13T11:54:39.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.774+0000 7f9ab64aa700 1 --2- 172.21.15.123:0/1398395379 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9a9802ff60 0x7f9a98032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:39.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.774+0000 7f9ab64aa700 1 --2- 172.21.15.123:0/1398395379 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ab00f5410 0x7f9ab0106710 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:39.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.775+0000 7f9ab64aa700 1 -- 172.21.15.123:0/1398395379 >> 172.21.15.123:0/1398395379 conn(0x7f9ab00f0b70 msgr2=0x7f9ab00f1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:39.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.775+0000 7f9ab64aa700 1 -- 172.21.15.123:0/1398395379 shutdown_connections 2024-01-13T11:54:39.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:39.775+0000 7f9ab64aa700 1 -- 172.21.15.123:0/1398395379 wait complete. 2024-01-13T11:54:39.763 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:40.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:39.641797+0000 mon.smithi059 (mon.0) 220 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/mirror_snapshot_schedule"}]: dispatch 2024-01-13T11:54:40.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:39.642793+0000 mon.smithi059 (mon.0) 221 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/trash_purge_schedule"}]: dispatch 2024-01-13T11:54:40.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:39.772967+0000 mon.smithi059 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.123:0/1398395379' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:41.162 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:41.162 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:41.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:41 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:39.547589+0000 mgr.smithi059.hmyldj (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:42.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.256+0000 7f68deb0a700 1 -- 172.21.15.123:0/1050784523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68d80f5410 msgr2=0x7f68d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:42.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.256+0000 7f68deb0a700 1 --2- 172.21.15.123:0/1050784523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68d80f5410 0x7f68d80f5830 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f68cc009fb0 tx=0x7f68cc009b30).stop 2024-01-13T11:54:42.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.256+0000 7f68deb0a700 1 -- 172.21.15.123:0/1050784523 shutdown_connections 2024-01-13T11:54:42.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.256+0000 7f68deb0a700 1 --2- 172.21.15.123:0/1050784523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68d80f5410 0x7f68d80f5830 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:42.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.256+0000 7f68deb0a700 1 -- 172.21.15.123:0/1050784523 >> 172.21.15.123:0/1050784523 conn(0x7f68d80f0b70 msgr2=0x7f68d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:42.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.256+0000 7f68deb0a700 1 -- 172.21.15.123:0/1050784523 shutdown_connections 2024-01-13T11:54:42.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.256+0000 7f68deb0a700 1 -- 172.21.15.123:0/1050784523 wait complete. 2024-01-13T11:54:42.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.257+0000 7f68deb0a700 1 Processor -- start 2024-01-13T11:54:42.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.257+0000 7f68deb0a700 1 -- start start 2024-01-13T11:54:42.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.257+0000 7f68deb0a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68d80f5410 0x7f68d8106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:42.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.257+0000 7f68deb0a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d8106c00 con 0x7f68d80f5410 2024-01-13T11:54:42.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.257+0000 7f68dc8a6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68d80f5410 0x7f68d8106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:42.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.257+0000 7f68dc8a6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68d80f5410 0x7f68d8106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:44668/0 (socket says 172.21.15.123:44668) 2024-01-13T11:54:42.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.257+0000 7f68dc8a6700 1 -- 172.21.15.123:0/1652986998 learned_addr learned my addr 172.21.15.123:0/1652986998 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:42.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.257+0000 7f68dc8a6700 1 -- 172.21.15.123:0/1652986998 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68cc005040 con 0x7f68d80f5410 2024-01-13T11:54:42.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.257+0000 7f68dc8a6700 1 --2- 172.21.15.123:0/1652986998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68d80f5410 0x7f68d8106710 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f68cc013510 tx=0x7f68cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:42.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.258+0000 7f68d57fa700 1 -- 172.21.15.123:0/1652986998 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68cc009320 con 0x7f68d80f5410 2024-01-13T11:54:42.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.258+0000 7f68d57fa700 1 -- 172.21.15.123:0/1652986998 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68cc013f00 con 0x7f68d80f5410 2024-01-13T11:54:42.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.258+0000 7f68deb0a700 1 -- 172.21.15.123:0/1652986998 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d8106e00 con 0x7f68d80f5410 2024-01-13T11:54:42.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.258+0000 7f68d57fa700 1 -- 172.21.15.123:0/1652986998 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68cc014d90 con 0x7f68d80f5410 2024-01-13T11:54:42.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.258+0000 7f68deb0a700 1 -- 172.21.15.123:0/1652986998 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d8107290 con 0x7f68d80f5410 2024-01-13T11:54:42.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.258+0000 7f68d57fa700 1 -- 172.21.15.123:0/1652986998 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68cc0263a0 con 0x7f68d80f5410 2024-01-13T11:54:42.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.259+0000 7f68d57fa700 1 --2- 172.21.15.123:0/1652986998 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f68c802ff60 0x7f68c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:42.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.259+0000 7f68d57fa700 1 -- 172.21.15.123:0/1652986998 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f68cc041010 con 0x7f68d80f5410 2024-01-13T11:54:42.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.259+0000 7f68deb0a700 1 -- 172.21.15.123:0/1652986998 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68d8045ad0 con 0x7f68d80f5410 2024-01-13T11:54:42.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.259+0000 7f68d7fff700 1 --2- 172.21.15.123:0/1652986998 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f68c802ff60 0x7f68c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:42.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.259+0000 7f68d7fff700 1 --2- 172.21.15.123:0/1652986998 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f68c802ff60 0x7f68c8032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f68c40096a0 tx=0x7f68c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:42.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.264+0000 7f68d57fa700 1 -- 172.21.15.123:0/1652986998 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f68cc012920 con 0x7f68d80f5410 2024-01-13T11:54:42.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.471+0000 7f68deb0a700 1 -- 172.21.15.123:0/1652986998 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68d80faf50 con 0x7f68d80f5410 2024-01-13T11:54:42.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.471+0000 7f68d57fa700 1 -- 172.21.15.123:0/1652986998 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f68cc02b020 con 0x7f68d80f5410 2024-01-13T11:54:42.459 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:42.459 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:42.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.473+0000 7f68deb0a700 1 -- 172.21.15.123:0/1652986998 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f68c802ff60 msgr2=0x7f68c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:42.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.473+0000 7f68deb0a700 1 --2- 172.21.15.123:0/1652986998 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f68c802ff60 0x7f68c8032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f68c40096a0 tx=0x7f68c4006b40).stop 2024-01-13T11:54:42.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.473+0000 7f68deb0a700 1 -- 172.21.15.123:0/1652986998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68d80f5410 msgr2=0x7f68d8106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:42.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.473+0000 7f68deb0a700 1 --2- 172.21.15.123:0/1652986998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68d80f5410 0x7f68d8106710 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f68cc013510 tx=0x7f68cc00b040).stop 2024-01-13T11:54:42.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.473+0000 7f68deb0a700 1 -- 172.21.15.123:0/1652986998 shutdown_connections 2024-01-13T11:54:42.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.473+0000 7f68deb0a700 1 --2- 172.21.15.123:0/1652986998 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f68c802ff60 0x7f68c8032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:42.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.473+0000 7f68deb0a700 1 --2- 172.21.15.123:0/1652986998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68d80f5410 0x7f68d8106710 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:42.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.473+0000 7f68deb0a700 1 -- 172.21.15.123:0/1652986998 >> 172.21.15.123:0/1652986998 conn(0x7f68d80f0b70 msgr2=0x7f68d805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:42.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.473+0000 7f68deb0a700 1 -- 172.21.15.123:0/1652986998 shutdown_connections 2024-01-13T11:54:42.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:42.473+0000 7f68deb0a700 1 -- 172.21.15.123:0/1652986998 wait complete. 2024-01-13T11:54:42.461 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:42.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:42 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:41.547861+0000 mgr.smithi059.hmyldj (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:43.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:43 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:42.471763+0000 mon.smithi059 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.123:0/1652986998' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:43.859 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:43.860 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:44.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:44 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:43.548147+0000 mgr.smithi059.hmyldj (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:44.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.982+0000 7f78d3430700 1 -- 172.21.15.123:0/3401446377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78cc0f5410 msgr2=0x7f78cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:44.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.982+0000 7f78d3430700 1 --2- 172.21.15.123:0/3401446377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78cc0f5410 0x7f78cc0f5830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f78c8009fb0 tx=0x7f78c8009b30).stop 2024-01-13T11:54:44.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.982+0000 7f78d3430700 1 -- 172.21.15.123:0/3401446377 shutdown_connections 2024-01-13T11:54:44.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.982+0000 7f78d3430700 1 --2- 172.21.15.123:0/3401446377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78cc0f5410 0x7f78cc0f5830 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:44.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.982+0000 7f78d3430700 1 -- 172.21.15.123:0/3401446377 >> 172.21.15.123:0/3401446377 conn(0x7f78cc0f0b70 msgr2=0x7f78cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:44.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.982+0000 7f78d3430700 1 -- 172.21.15.123:0/3401446377 shutdown_connections 2024-01-13T11:54:44.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.982+0000 7f78d3430700 1 -- 172.21.15.123:0/3401446377 wait complete. 2024-01-13T11:54:44.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.982+0000 7f78d3430700 1 Processor -- start 2024-01-13T11:54:44.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.982+0000 7f78d3430700 1 -- start start 2024-01-13T11:54:44.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.983+0000 7f78d3430700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78cc0f5410 0x7f78cc109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:44.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.983+0000 7f78d3430700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc060240 con 0x7f78cc0f5410 2024-01-13T11:54:44.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.983+0000 7f78d11cc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78cc0f5410 0x7f78cc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:44.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.983+0000 7f78d11cc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78cc0f5410 0x7f78cc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:44674/0 (socket says 172.21.15.123:44674) 2024-01-13T11:54:44.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.983+0000 7f78d11cc700 1 -- 172.21.15.123:0/2337711026 learned_addr learned my addr 172.21.15.123:0/2337711026 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:44.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.983+0000 7f78d11cc700 1 -- 172.21.15.123:0/2337711026 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c8005040 con 0x7f78cc0f5410 2024-01-13T11:54:44.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.983+0000 7f78d11cc700 1 --2- 172.21.15.123:0/2337711026 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78cc0f5410 0x7f78cc109430 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f78c8013510 tx=0x7f78c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:44.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.983+0000 7f78c1ffb700 1 -- 172.21.15.123:0/2337711026 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78c8009320 con 0x7f78cc0f5410 2024-01-13T11:54:44.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.983+0000 7f78d3430700 1 -- 172.21.15.123:0/2337711026 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78cc060440 con 0x7f78cc0f5410 2024-01-13T11:54:44.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.983+0000 7f78c1ffb700 1 -- 172.21.15.123:0/2337711026 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f78c8013f00 con 0x7f78cc0f5410 2024-01-13T11:54:44.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.984+0000 7f78c1ffb700 1 -- 172.21.15.123:0/2337711026 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78c8014d90 con 0x7f78cc0f5410 2024-01-13T11:54:44.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.984+0000 7f78d3430700 1 -- 172.21.15.123:0/2337711026 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78cc109ae0 con 0x7f78cc0f5410 2024-01-13T11:54:44.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.984+0000 7f78c1ffb700 1 -- 172.21.15.123:0/2337711026 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f78c8014420 con 0x7f78cc0f5410 2024-01-13T11:54:44.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.984+0000 7f78d3430700 1 -- 172.21.15.123:0/2337711026 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78cc045ad0 con 0x7f78cc0f5410 2024-01-13T11:54:44.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.985+0000 7f78c1ffb700 1 --2- 172.21.15.123:0/2337711026 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f78b802ff60 0x7f78b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:44.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.985+0000 7f78c1ffb700 1 -- 172.21.15.123:0/2337711026 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f78c8041110 con 0x7f78cc0f5410 2024-01-13T11:54:44.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.985+0000 7f78d09cb700 1 --2- 172.21.15.123:0/2337711026 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f78b802ff60 0x7f78b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:44.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.985+0000 7f78d09cb700 1 --2- 172.21.15.123:0/2337711026 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f78b802ff60 0x7f78b8032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f78bc0096a0 tx=0x7f78bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:44.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:44.990+0000 7f78c1ffb700 1 -- 172.21.15.123:0/2337711026 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f78c80095f0 con 0x7f78cc0f5410 2024-01-13T11:54:45.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:45.198+0000 7f78d3430700 1 -- 172.21.15.123:0/2337711026 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78cc0fae60 con 0x7f78cc0f5410 2024-01-13T11:54:45.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:45.199+0000 7f78c1ffb700 1 -- 172.21.15.123:0/2337711026 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f78c801db70 con 0x7f78cc0f5410 2024-01-13T11:54:45.188 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:45.188 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:45.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:45.201+0000 7f78d3430700 1 -- 172.21.15.123:0/2337711026 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f78b802ff60 msgr2=0x7f78b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:45.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:45.201+0000 7f78d3430700 1 --2- 172.21.15.123:0/2337711026 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f78b802ff60 0x7f78b8032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f78bc0096a0 tx=0x7f78bc006b40).stop 2024-01-13T11:54:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:45.201+0000 7f78d3430700 1 -- 172.21.15.123:0/2337711026 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78cc0f5410 msgr2=0x7f78cc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:45.201+0000 7f78d3430700 1 --2- 172.21.15.123:0/2337711026 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78cc0f5410 0x7f78cc109430 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f78c8013510 tx=0x7f78c800b040).stop 2024-01-13T11:54:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:45.201+0000 7f78d3430700 1 -- 172.21.15.123:0/2337711026 shutdown_connections 2024-01-13T11:54:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:45.201+0000 7f78d3430700 1 --2- 172.21.15.123:0/2337711026 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f78b802ff60 0x7f78b8032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:45.201+0000 7f78d3430700 1 --2- 172.21.15.123:0/2337711026 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78cc0f5410 0x7f78cc109430 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:45.201+0000 7f78d3430700 1 -- 172.21.15.123:0/2337711026 >> 172.21.15.123:0/2337711026 conn(0x7f78cc0f0b70 msgr2=0x7f78cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:45.201+0000 7f78d3430700 1 -- 172.21.15.123:0/2337711026 shutdown_connections 2024-01-13T11:54:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:45.201+0000 7f78d3430700 1 -- 172.21.15.123:0/2337711026 wait complete. 2024-01-13T11:54:45.190 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:45.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:45 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:45.200576+0000 mon.smithi059 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.123:0/2337711026' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:46.581 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:46.581 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:47.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:47 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:45.548510+0000 mgr.smithi059.hmyldj (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:47.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.764+0000 7fc4154f0700 1 -- 172.21.15.123:0/2590484203 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4100f6510 msgr2=0x7fc4100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:47.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.764+0000 7fc4154f0700 1 --2- 172.21.15.123:0/2590484203 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4100f6510 0x7fc4100f6930 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fc400009fb0 tx=0x7fc400009b30).stop 2024-01-13T11:54:47.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.764+0000 7fc4154f0700 1 -- 172.21.15.123:0/2590484203 shutdown_connections 2024-01-13T11:54:47.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.764+0000 7fc4154f0700 1 --2- 172.21.15.123:0/2590484203 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4100f6510 0x7fc4100f6930 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:47.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.764+0000 7fc4154f0700 1 -- 172.21.15.123:0/2590484203 >> 172.21.15.123:0/2590484203 conn(0x7fc4100f1c70 msgr2=0x7fc4100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:47.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.764+0000 7fc4154f0700 1 -- 172.21.15.123:0/2590484203 shutdown_connections 2024-01-13T11:54:47.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.764+0000 7fc4154f0700 1 -- 172.21.15.123:0/2590484203 wait complete. 2024-01-13T11:54:47.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.765+0000 7fc4154f0700 1 Processor -- start 2024-01-13T11:54:47.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.765+0000 7fc4154f0700 1 -- start start 2024-01-13T11:54:47.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.765+0000 7fc4154f0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4100f6510 0x7fc410106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:47.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.765+0000 7fc4154f0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc410106c20 con 0x7fc4100f6510 2024-01-13T11:54:47.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.765+0000 7fc40effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4100f6510 0x7fc410106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:47.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.765+0000 7fc40effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4100f6510 0x7fc410106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:38780/0 (socket says 172.21.15.123:38780) 2024-01-13T11:54:47.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.765+0000 7fc40effd700 1 -- 172.21.15.123:0/1250142350 learned_addr learned my addr 172.21.15.123:0/1250142350 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:47.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.766+0000 7fc40effd700 1 -- 172.21.15.123:0/1250142350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc400005040 con 0x7fc4100f6510 2024-01-13T11:54:47.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.766+0000 7fc40effd700 1 --2- 172.21.15.123:0/1250142350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4100f6510 0x7fc410106730 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fc400013510 tx=0x7fc40000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:47.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.766+0000 7fc3f77fe700 1 -- 172.21.15.123:0/1250142350 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc400009320 con 0x7fc4100f6510 2024-01-13T11:54:47.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.766+0000 7fc4154f0700 1 -- 172.21.15.123:0/1250142350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc410106e20 con 0x7fc4100f6510 2024-01-13T11:54:47.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.766+0000 7fc3f77fe700 1 -- 172.21.15.123:0/1250142350 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc400013f00 con 0x7fc4100f6510 2024-01-13T11:54:47.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.766+0000 7fc3f77fe700 1 -- 172.21.15.123:0/1250142350 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc400014d90 con 0x7fc4100f6510 2024-01-13T11:54:47.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.766+0000 7fc4154f0700 1 -- 172.21.15.123:0/1250142350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4101072b0 con 0x7fc4100f6510 2024-01-13T11:54:47.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.767+0000 7fc3f77fe700 1 -- 172.21.15.123:0/1250142350 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc400014420 con 0x7fc4100f6510 2024-01-13T11:54:47.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.767+0000 7fc4154f0700 1 -- 172.21.15.123:0/1250142350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc410045ad0 con 0x7fc4100f6510 2024-01-13T11:54:47.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.767+0000 7fc3f77fe700 1 --2- 172.21.15.123:0/1250142350 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc3f802ff60 0x7fc3f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:47.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.767+0000 7fc3f77fe700 1 -- 172.21.15.123:0/1250142350 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc400041000 con 0x7fc4100f6510 2024-01-13T11:54:47.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.767+0000 7fc40e7fc700 1 --2- 172.21.15.123:0/1250142350 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc3f802ff60 0x7fc3f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:47.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.768+0000 7fc40e7fc700 1 --2- 172.21.15.123:0/1250142350 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc3f802ff60 0x7fc3f8032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc4040096a0 tx=0x7fc404006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:47.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.772+0000 7fc3f77fe700 1 -- 172.21.15.123:0/1250142350 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4000095f0 con 0x7fc4100f6510 2024-01-13T11:54:47.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.978+0000 7fc4154f0700 1 -- 172.21.15.123:0/1250142350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4100fbf60 con 0x7fc4100f6510 2024-01-13T11:54:47.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.978+0000 7fc3f77fe700 1 -- 172.21.15.123:0/1250142350 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc40001db70 con 0x7fc4100f6510 2024-01-13T11:54:47.967 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:47.967 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:47.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.980+0000 7fc4154f0700 1 -- 172.21.15.123:0/1250142350 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc3f802ff60 msgr2=0x7fc3f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:47.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.980+0000 7fc4154f0700 1 --2- 172.21.15.123:0/1250142350 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc3f802ff60 0x7fc3f8032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc4040096a0 tx=0x7fc404006b40).stop 2024-01-13T11:54:47.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.980+0000 7fc4154f0700 1 -- 172.21.15.123:0/1250142350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4100f6510 msgr2=0x7fc410106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:47.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.980+0000 7fc4154f0700 1 --2- 172.21.15.123:0/1250142350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4100f6510 0x7fc410106730 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fc400013510 tx=0x7fc40000b040).stop 2024-01-13T11:54:47.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.980+0000 7fc4154f0700 1 -- 172.21.15.123:0/1250142350 shutdown_connections 2024-01-13T11:54:47.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.980+0000 7fc4154f0700 1 --2- 172.21.15.123:0/1250142350 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc3f802ff60 0x7fc3f8032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:47.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.980+0000 7fc4154f0700 1 --2- 172.21.15.123:0/1250142350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4100f6510 0x7fc410106730 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:47.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.980+0000 7fc4154f0700 1 -- 172.21.15.123:0/1250142350 >> 172.21.15.123:0/1250142350 conn(0x7fc4100f1c70 msgr2=0x7fc4100f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:47.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.980+0000 7fc4154f0700 1 -- 172.21.15.123:0/1250142350 shutdown_connections 2024-01-13T11:54:47.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:47.980+0000 7fc4154f0700 1 -- 172.21.15.123:0/1250142350 wait complete. 2024-01-13T11:54:47.969 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:48.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:48 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:47.979774+0000 mon.smithi059 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.123:0/1250142350' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:49.371 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:49.372 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:49.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:49 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:47.548828+0000 mgr.smithi059.hmyldj (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:50.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.483+0000 7fb811a2e700 1 -- 172.21.15.123:0/1908367542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c0f5410 msgr2=0x7fb80c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:50.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.483+0000 7fb811a2e700 1 --2- 172.21.15.123:0/1908367542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c0f5410 0x7fb80c0f5830 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fb800009fb0 tx=0x7fb800009b30).stop 2024-01-13T11:54:50.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.483+0000 7fb811a2e700 1 -- 172.21.15.123:0/1908367542 shutdown_connections 2024-01-13T11:54:50.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.483+0000 7fb811a2e700 1 --2- 172.21.15.123:0/1908367542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c0f5410 0x7fb80c0f5830 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:50.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.483+0000 7fb811a2e700 1 -- 172.21.15.123:0/1908367542 >> 172.21.15.123:0/1908367542 conn(0x7fb80c0f0b70 msgr2=0x7fb80c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:50.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.483+0000 7fb811a2e700 1 -- 172.21.15.123:0/1908367542 shutdown_connections 2024-01-13T11:54:50.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.483+0000 7fb811a2e700 1 -- 172.21.15.123:0/1908367542 wait complete. 2024-01-13T11:54:50.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.484+0000 7fb811a2e700 1 Processor -- start 2024-01-13T11:54:50.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.484+0000 7fb811a2e700 1 -- start start 2024-01-13T11:54:50.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.484+0000 7fb811a2e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c0f5410 0x7fb80c106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:50.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.484+0000 7fb811a2e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c106c50 con 0x7fb80c0f5410 2024-01-13T11:54:50.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.484+0000 7fb80affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c0f5410 0x7fb80c106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:50.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.484+0000 7fb80affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c0f5410 0x7fb80c106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:38808/0 (socket says 172.21.15.123:38808) 2024-01-13T11:54:50.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.484+0000 7fb80affd700 1 -- 172.21.15.123:0/4140870227 learned_addr learned my addr 172.21.15.123:0/4140870227 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:50.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.485+0000 7fb80affd700 1 -- 172.21.15.123:0/4140870227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb800005040 con 0x7fb80c0f5410 2024-01-13T11:54:50.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.485+0000 7fb80affd700 1 --2- 172.21.15.123:0/4140870227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c0f5410 0x7fb80c106760 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fb800009f80 tx=0x7fb80000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:50.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.485+0000 7fb7f37fe700 1 -- 172.21.15.123:0/4140870227 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb800009320 con 0x7fb80c0f5410 2024-01-13T11:54:50.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.485+0000 7fb811a2e700 1 -- 172.21.15.123:0/4140870227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb80c106e50 con 0x7fb80c0f5410 2024-01-13T11:54:50.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.485+0000 7fb7f37fe700 1 -- 172.21.15.123:0/4140870227 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb800013ec0 con 0x7fb80c0f5410 2024-01-13T11:54:50.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.485+0000 7fb7f37fe700 1 -- 172.21.15.123:0/4140870227 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb80001ece0 con 0x7fb80c0f5410 2024-01-13T11:54:50.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.485+0000 7fb811a2e700 1 -- 172.21.15.123:0/4140870227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80c1072e0 con 0x7fb80c0f5410 2024-01-13T11:54:50.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.486+0000 7fb7f37fe700 1 -- 172.21.15.123:0/4140870227 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb80001e420 con 0x7fb80c0f5410 2024-01-13T11:54:50.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.486+0000 7fb811a2e700 1 -- 172.21.15.123:0/4140870227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb80c045ad0 con 0x7fb80c0f5410 2024-01-13T11:54:50.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.486+0000 7fb7f37fe700 1 --2- 172.21.15.123:0/4140870227 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb7f402ff60 0x7fb7f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:50.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.486+0000 7fb7f37fe700 1 -- 172.21.15.123:0/4140870227 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb800040d10 con 0x7fb80c0f5410 2024-01-13T11:54:50.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.486+0000 7fb80a7fc700 1 --2- 172.21.15.123:0/4140870227 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb7f402ff60 0x7fb7f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:50.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.487+0000 7fb80a7fc700 1 --2- 172.21.15.123:0/4140870227 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb7f402ff60 0x7fb7f4032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fb7fc0096a0 tx=0x7fb7fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:50.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.491+0000 7fb7f37fe700 1 -- 172.21.15.123:0/4140870227 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb80001c920 con 0x7fb80c0f5410 2024-01-13T11:54:50.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.696+0000 7fb811a2e700 1 -- 172.21.15.123:0/4140870227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb80c108130 con 0x7fb80c0f5410 2024-01-13T11:54:50.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.697+0000 7fb7f37fe700 1 -- 172.21.15.123:0/4140870227 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb80002c2d0 con 0x7fb80c0f5410 2024-01-13T11:54:50.685 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:50.685 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:50.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.698+0000 7fb811a2e700 1 -- 172.21.15.123:0/4140870227 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb7f402ff60 msgr2=0x7fb7f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:50.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.698+0000 7fb811a2e700 1 --2- 172.21.15.123:0/4140870227 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb7f402ff60 0x7fb7f4032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fb7fc0096a0 tx=0x7fb7fc006b40).stop 2024-01-13T11:54:50.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.698+0000 7fb811a2e700 1 -- 172.21.15.123:0/4140870227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c0f5410 msgr2=0x7fb80c106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:50.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.698+0000 7fb811a2e700 1 --2- 172.21.15.123:0/4140870227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c0f5410 0x7fb80c106760 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fb800009f80 tx=0x7fb80000b040).stop 2024-01-13T11:54:50.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.698+0000 7fb811a2e700 1 -- 172.21.15.123:0/4140870227 shutdown_connections 2024-01-13T11:54:50.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.698+0000 7fb811a2e700 1 --2- 172.21.15.123:0/4140870227 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb7f402ff60 0x7fb7f4032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:50.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.698+0000 7fb811a2e700 1 --2- 172.21.15.123:0/4140870227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb80c0f5410 0x7fb80c106760 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:50.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.698+0000 7fb811a2e700 1 -- 172.21.15.123:0/4140870227 >> 172.21.15.123:0/4140870227 conn(0x7fb80c0f0b70 msgr2=0x7fb80c05fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:50.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.698+0000 7fb811a2e700 1 -- 172.21.15.123:0/4140870227 shutdown_connections 2024-01-13T11:54:50.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:50.698+0000 7fb811a2e700 1 -- 172.21.15.123:0/4140870227 wait complete. 2024-01-13T11:54:50.688 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:51.594 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:49.549162+0000 mgr.smithi059.hmyldj (mgr.14162) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:51.594 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:50.534351+0000 mon.smithi059 (mon.0) 226 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi059", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T11:54:51.594 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:50.539355+0000 mon.smithi059 (mon.0) 227 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi059", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-13T11:54:51.595 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:50.540216+0000 mon.smithi059 (mon.0) 228 : audit [DBG] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:54:51.595 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:50.698221+0000 mon.smithi059 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.123:0/4140870227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:52.061 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:52.061 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:52.387 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cephadm 2024-01-13T11:54:50.533137+0000 mgr.smithi059.hmyldj (mgr.14162) 157 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi059 ... 2024-01-13T11:54:52.387 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-13T11:54:52.388 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-13T11:54:52.388 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr time="2024-01-13T11:50:28Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-13T11:54:52.388 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr time="2024-01-13T11:51:06Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-13T11:54:52.388 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr time="2024-01-13T11:51:43Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-13T11:54:52.388 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-13T11:54:52.388 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-13T11:54:52.388 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-13T11:54:52.388 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr time="2024-01-13T11:52:57Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-13T11:54:52.388 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr time="2024-01-13T11:53:35Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-13T11:54:52.388 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr time="2024-01-13T11:54:12Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-13T11:54:52.388 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-13T11:54:52.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Traceback (most recent call last): 2024-01-13T11:54:52.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-13T11:54:52.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: main() 2024-01-13T11:54:52.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-13T11:54:52.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: r = ctx.func(ctx) 2024-01-13T11:54:52.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-13T11:54:52.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: return func(ctx) 2024-01-13T11:54:52.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-13T11:54:52.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-13T11:54:52.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-13T11:54:52.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-13T11:54:52.389 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-13T11:54:52.390 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: raise RuntimeError('uid/gid not found') 2024-01-13T11:54:52.390 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: RuntimeError: uid/gid not found 2024-01-13T11:54:52.390 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Traceback (most recent call last): 2024-01-13T11:54:52.390 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-13T11:54:52.390 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: yield (conn, connr) 2024-01-13T11:54:52.390 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-13T11:54:52.390 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: code, '\n'.join(err))) 2024-01-13T11:54:52.390 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi059 ... 2024-01-13T11:54:52.390 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-13T11:54:52.390 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-13T11:54:52.390 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr time="2024-01-13T11:50:28Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-13T11:54:52.390 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr time="2024-01-13T11:51:06Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-13T11:54:52.390 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr time="2024-01-13T11:51:43Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-13T11:54:52.391 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-13T11:54:52.391 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-13T11:54:52.391 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-13T11:54:52.391 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr time="2024-01-13T11:52:57Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-13T11:54:52.391 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr time="2024-01-13T11:53:35Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-13T11:54:52.391 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr time="2024-01-13T11:54:12Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-13T11:54:52.392 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-13T11:54:52.392 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Traceback (most recent call last): 2024-01-13T11:54:52.392 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-13T11:54:52.392 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: main() 2024-01-13T11:54:52.392 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-13T11:54:52.392 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: r = ctx.func(ctx) 2024-01-13T11:54:52.392 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-13T11:54:52.392 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: return func(ctx) 2024-01-13T11:54:52.392 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-13T11:54:52.392 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-13T11:54:52.392 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-13T11:54:52.392 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-13T11:54:52.392 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-13T11:54:52.393 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: raise RuntimeError('uid/gid not found') 2024-01-13T11:54:52.393 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: RuntimeError: uid/gid not found 2024-01-13T11:54:52.393 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cephadm 2024-01-13T11:54:50.541116+0000 mgr.smithi059.hmyldj (mgr.14162) 158 : cephadm [INF] Deploying daemon crash.smithi059 on smithi059 2024-01-13T11:54:52.393 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:52.167988+0000 mon.smithi059 (mon.0) 230 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' 2024-01-13T11:54:53.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.207+0000 7fc5f9533700 1 -- 172.21.15.123:0/4076219180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5f40f5410 msgr2=0x7fc5f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.207+0000 7fc5f9533700 1 --2- 172.21.15.123:0/4076219180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5f40f5410 0x7fc5f40f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fc5e8009fb0 tx=0x7fc5e8009b30).stop 2024-01-13T11:54:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.207+0000 7fc5f9533700 1 -- 172.21.15.123:0/4076219180 shutdown_connections 2024-01-13T11:54:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.207+0000 7fc5f9533700 1 --2- 172.21.15.123:0/4076219180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5f40f5410 0x7fc5f40f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.207+0000 7fc5f9533700 1 -- 172.21.15.123:0/4076219180 >> 172.21.15.123:0/4076219180 conn(0x7fc5f40f0b70 msgr2=0x7fc5f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.207+0000 7fc5f9533700 1 -- 172.21.15.123:0/4076219180 shutdown_connections 2024-01-13T11:54:53.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.207+0000 7fc5f9533700 1 -- 172.21.15.123:0/4076219180 wait complete. 2024-01-13T11:54:53.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.208+0000 7fc5f9533700 1 Processor -- start 2024-01-13T11:54:53.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.208+0000 7fc5f9533700 1 -- start start 2024-01-13T11:54:53.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.208+0000 7fc5f9533700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5f40f5410 0x7fc5f4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:53.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.208+0000 7fc5f9533700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f4105b20 con 0x7fc5f40f5410 2024-01-13T11:54:53.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.208+0000 7fc5f2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5f40f5410 0x7fc5f4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:53.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.208+0000 7fc5f2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5f40f5410 0x7fc5f4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:38836/0 (socket says 172.21.15.123:38836) 2024-01-13T11:54:53.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.208+0000 7fc5f2ffd700 1 -- 172.21.15.123:0/1758902622 learned_addr learned my addr 172.21.15.123:0/1758902622 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:53.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.208+0000 7fc5f2ffd700 1 -- 172.21.15.123:0/1758902622 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e8005040 con 0x7fc5f40f5410 2024-01-13T11:54:53.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.209+0000 7fc5f2ffd700 1 --2- 172.21.15.123:0/1758902622 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5f40f5410 0x7fc5f4105630 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fc5e8013510 tx=0x7fc5e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:53.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.209+0000 7fc5db7fe700 1 -- 172.21.15.123:0/1758902622 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5e8009320 con 0x7fc5f40f5410 2024-01-13T11:54:53.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.209+0000 7fc5f9533700 1 -- 172.21.15.123:0/1758902622 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f4105d20 con 0x7fc5f40f5410 2024-01-13T11:54:53.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.209+0000 7fc5db7fe700 1 -- 172.21.15.123:0/1758902622 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5e8013f00 con 0x7fc5f40f5410 2024-01-13T11:54:53.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.209+0000 7fc5db7fe700 1 -- 172.21.15.123:0/1758902622 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5e8014d90 con 0x7fc5f40f5410 2024-01-13T11:54:53.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.209+0000 7fc5f9533700 1 -- 172.21.15.123:0/1758902622 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f41061b0 con 0x7fc5f40f5410 2024-01-13T11:54:53.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.210+0000 7fc5db7fe700 1 -- 172.21.15.123:0/1758902622 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5e80263a0 con 0x7fc5f40f5410 2024-01-13T11:54:53.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.210+0000 7fc5db7fe700 1 --2- 172.21.15.123:0/1758902622 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc5dc02ff60 0x7fc5dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:53.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.210+0000 7fc5db7fe700 1 -- 172.21.15.123:0/1758902622 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc5e80410a0 con 0x7fc5f40f5410 2024-01-13T11:54:53.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.210+0000 7fc5f9533700 1 -- 172.21.15.123:0/1758902622 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f4045ad0 con 0x7fc5f40f5410 2024-01-13T11:54:53.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.210+0000 7fc5f27fc700 1 --2- 172.21.15.123:0/1758902622 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc5dc02ff60 0x7fc5dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:53.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.210+0000 7fc5f27fc700 1 --2- 172.21.15.123:0/1758902622 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc5dc02ff60 0x7fc5dc032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fc5e40096a0 tx=0x7fc5e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:53.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.215+0000 7fc5db7fe700 1 -- 172.21.15.123:0/1758902622 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc5e8012920 con 0x7fc5f40f5410 2024-01-13T11:54:53.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.421+0000 7fc5f9533700 1 -- 172.21.15.123:0/1758902622 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc5f4106ed0 con 0x7fc5f40f5410 2024-01-13T11:54:53.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.422+0000 7fc5db7fe700 1 -- 172.21.15.123:0/1758902622 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc5e802b020 con 0x7fc5f40f5410 2024-01-13T11:54:53.410 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:53.410 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:53.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.423+0000 7fc5f9533700 1 -- 172.21.15.123:0/1758902622 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc5dc02ff60 msgr2=0x7fc5dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:53.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.423+0000 7fc5f9533700 1 --2- 172.21.15.123:0/1758902622 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc5dc02ff60 0x7fc5dc032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fc5e40096a0 tx=0x7fc5e4006b40).stop 2024-01-13T11:54:53.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.423+0000 7fc5f9533700 1 -- 172.21.15.123:0/1758902622 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5f40f5410 msgr2=0x7fc5f4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:53.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.423+0000 7fc5f9533700 1 --2- 172.21.15.123:0/1758902622 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5f40f5410 0x7fc5f4105630 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fc5e8013510 tx=0x7fc5e800b040).stop 2024-01-13T11:54:53.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.423+0000 7fc5f9533700 1 -- 172.21.15.123:0/1758902622 shutdown_connections 2024-01-13T11:54:53.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.423+0000 7fc5f9533700 1 --2- 172.21.15.123:0/1758902622 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc5dc02ff60 0x7fc5dc032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:53.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.423+0000 7fc5f9533700 1 --2- 172.21.15.123:0/1758902622 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5f40f5410 0x7fc5f4105630 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:53.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.423+0000 7fc5f9533700 1 -- 172.21.15.123:0/1758902622 >> 172.21.15.123:0/1758902622 conn(0x7fc5f40f0b70 msgr2=0x7fc5f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:53.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.423+0000 7fc5f9533700 1 -- 172.21.15.123:0/1758902622 shutdown_connections 2024-01-13T11:54:53.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:53.423+0000 7fc5f9533700 1 -- 172.21.15.123:0/1758902622 wait complete. 2024-01-13T11:54:53.413 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:53.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:53 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:51.549423+0000 mgr.smithi059.hmyldj (mgr.14162) 159 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:53.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:53 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:52.326682+0000 mon.smithi059 (mon.0) 231 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' 2024-01-13T11:54:53.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:53 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:52.328595+0000 mon.smithi059 (mon.0) 232 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' 2024-01-13T11:54:53.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:53 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:52.329001+0000 mon.smithi059 (mon.0) 233 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-13T11:54:53.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:53 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:52.331835+0000 mon.smithi059 (mon.0) 234 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' 2024-01-13T11:54:54.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:54 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:52.329448+0000 mgr.smithi059.hmyldj (mgr.14162) 160 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-13T11:54:54.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:54 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cephadm 2024-01-13T11:54:52.334819+0000 mgr.smithi059.hmyldj (mgr.14162) 161 : cephadm [INF] Deploying daemon grafana.smithi059 on smithi059 2024-01-13T11:54:54.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:54 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:53.422913+0000 mon.smithi059 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.123:0/1758902622' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:54.792 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:54.792 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:55.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:55 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:53.549768+0000 mgr.smithi059.hmyldj (mgr.14162) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:55.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.936+0000 7fc35345b700 1 -- 172.21.15.123:0/364055367 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc34c0f6480 msgr2=0x7fc34c0f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:55.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.936+0000 7fc35345b700 1 --2- 172.21.15.123:0/364055367 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc34c0f6480 0x7fc34c0f68a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fc33c009fb0 tx=0x7fc33c009b30).stop 2024-01-13T11:54:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.936+0000 7fc35345b700 1 -- 172.21.15.123:0/364055367 shutdown_connections 2024-01-13T11:54:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.936+0000 7fc35345b700 1 --2- 172.21.15.123:0/364055367 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc34c0f6480 0x7fc34c0f68a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.936+0000 7fc35345b700 1 -- 172.21.15.123:0/364055367 >> 172.21.15.123:0/364055367 conn(0x7fc34c0f1c00 msgr2=0x7fc34c0f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.936+0000 7fc35345b700 1 -- 172.21.15.123:0/364055367 shutdown_connections 2024-01-13T11:54:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.936+0000 7fc35345b700 1 -- 172.21.15.123:0/364055367 wait complete. 2024-01-13T11:54:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.937+0000 7fc35345b700 1 Processor -- start 2024-01-13T11:54:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.937+0000 7fc35345b700 1 -- start start 2024-01-13T11:54:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.937+0000 7fc35345b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc34c0f6480 0x7fc34c1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.937+0000 7fc35345b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c106bc0 con 0x7fc34c0f6480 2024-01-13T11:54:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.937+0000 7fc3511f7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc34c0f6480 0x7fc34c1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.937+0000 7fc3511f7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc34c0f6480 0x7fc34c1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:38854/0 (socket says 172.21.15.123:38854) 2024-01-13T11:54:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.937+0000 7fc3511f7700 1 -- 172.21.15.123:0/2315885387 learned_addr learned my addr 172.21.15.123:0/2315885387 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.937+0000 7fc3511f7700 1 -- 172.21.15.123:0/2315885387 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33c005040 con 0x7fc34c0f6480 2024-01-13T11:54:55.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.938+0000 7fc3511f7700 1 --2- 172.21.15.123:0/2315885387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc34c0f6480 0x7fc34c1066d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fc33c013510 tx=0x7fc33c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:55.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.938+0000 7fc341ffb700 1 -- 172.21.15.123:0/2315885387 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc33c009320 con 0x7fc34c0f6480 2024-01-13T11:54:55.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.938+0000 7fc35345b700 1 -- 172.21.15.123:0/2315885387 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc34c106dc0 con 0x7fc34c0f6480 2024-01-13T11:54:55.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.938+0000 7fc341ffb700 1 -- 172.21.15.123:0/2315885387 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc33c013f00 con 0x7fc34c0f6480 2024-01-13T11:54:55.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.938+0000 7fc341ffb700 1 -- 172.21.15.123:0/2315885387 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc33c014d90 con 0x7fc34c0f6480 2024-01-13T11:54:55.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.938+0000 7fc35345b700 1 -- 172.21.15.123:0/2315885387 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc34c107250 con 0x7fc34c0f6480 2024-01-13T11:54:55.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.939+0000 7fc341ffb700 1 -- 172.21.15.123:0/2315885387 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc33c0095f0 con 0x7fc34c0f6480 2024-01-13T11:54:55.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.939+0000 7fc35345b700 1 -- 172.21.15.123:0/2315885387 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc34c045ad0 con 0x7fc34c0f6480 2024-01-13T11:54:55.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.939+0000 7fc341ffb700 1 --2- 172.21.15.123:0/2315885387 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc33802ff60 0x7fc338032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:55.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.939+0000 7fc341ffb700 1 -- 172.21.15.123:0/2315885387 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc33c041000 con 0x7fc34c0f6480 2024-01-13T11:54:55.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.939+0000 7fc3509f6700 1 --2- 172.21.15.123:0/2315885387 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc33802ff60 0x7fc338032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:55.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.940+0000 7fc3509f6700 1 --2- 172.21.15.123:0/2315885387 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc33802ff60 0x7fc338032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fc3480096a0 tx=0x7fc348006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:55.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:55.944+0000 7fc341ffb700 1 -- 172.21.15.123:0/2315885387 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc33c01d7f0 con 0x7fc34c0f6480 2024-01-13T11:54:56.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:56.152+0000 7fc35345b700 1 -- 172.21.15.123:0/2315885387 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc34c108100 con 0x7fc34c0f6480 2024-01-13T11:54:56.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:56.152+0000 7fc341ffb700 1 -- 172.21.15.123:0/2315885387 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc33c010230 con 0x7fc34c0f6480 2024-01-13T11:54:56.141 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:56.141 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:56.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:56.154+0000 7fc35345b700 1 -- 172.21.15.123:0/2315885387 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc33802ff60 msgr2=0x7fc338032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:56.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:56.154+0000 7fc35345b700 1 --2- 172.21.15.123:0/2315885387 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc33802ff60 0x7fc338032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fc3480096a0 tx=0x7fc348006b40).stop 2024-01-13T11:54:56.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:56.154+0000 7fc35345b700 1 -- 172.21.15.123:0/2315885387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc34c0f6480 msgr2=0x7fc34c1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:56.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:56.154+0000 7fc35345b700 1 --2- 172.21.15.123:0/2315885387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc34c0f6480 0x7fc34c1066d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fc33c013510 tx=0x7fc33c00b040).stop 2024-01-13T11:54:56.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:56.154+0000 7fc35345b700 1 -- 172.21.15.123:0/2315885387 shutdown_connections 2024-01-13T11:54:56.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:56.154+0000 7fc35345b700 1 --2- 172.21.15.123:0/2315885387 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc33802ff60 0x7fc338032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:56.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:56.154+0000 7fc35345b700 1 --2- 172.21.15.123:0/2315885387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc34c0f6480 0x7fc34c1066d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:56.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:56.154+0000 7fc35345b700 1 -- 172.21.15.123:0/2315885387 >> 172.21.15.123:0/2315885387 conn(0x7fc34c0f1c00 msgr2=0x7fc34c0f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:56.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:56.154+0000 7fc35345b700 1 -- 172.21.15.123:0/2315885387 shutdown_connections 2024-01-13T11:54:56.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:56.154+0000 7fc35345b700 1 -- 172.21.15.123:0/2315885387 wait complete. 2024-01-13T11:54:56.143 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:56.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:56 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:56.153533+0000 mon.smithi059 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.123:0/2315885387' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:54:57.544 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:54:57.544 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:54:57.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:55.550033+0000 mgr.smithi059.hmyldj (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:58.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.761+0000 7f788f73c700 1 -- 172.21.15.123:0/412384333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78880601e0 msgr2=0x7f78880f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:58.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.761+0000 7f788f73c700 1 --2- 172.21.15.123:0/412384333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78880601e0 0x7f78880f9190 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f7884009fb0 tx=0x7f7884009b30).stop 2024-01-13T11:54:58.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.761+0000 7f788f73c700 1 -- 172.21.15.123:0/412384333 shutdown_connections 2024-01-13T11:54:58.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.761+0000 7f788f73c700 1 --2- 172.21.15.123:0/412384333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78880601e0 0x7f78880f9190 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:58.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.761+0000 7f788f73c700 1 -- 172.21.15.123:0/412384333 >> 172.21.15.123:0/412384333 conn(0x7f78880f0b70 msgr2=0x7f78880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:58.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.761+0000 7f788f73c700 1 -- 172.21.15.123:0/412384333 shutdown_connections 2024-01-13T11:54:58.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.762+0000 7f788f73c700 1 -- 172.21.15.123:0/412384333 wait complete. 2024-01-13T11:54:58.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.762+0000 7f788f73c700 1 Processor -- start 2024-01-13T11:54:58.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.762+0000 7f788f73c700 1 -- start start 2024-01-13T11:54:58.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.762+0000 7f788f73c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78880601e0 0x7f78880f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:58.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.762+0000 7f788f73c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78880f5090 con 0x7f78880601e0 2024-01-13T11:54:58.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.763+0000 7f788d4d8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78880601e0 0x7f78880f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.763+0000 7f788d4d8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78880601e0 0x7f78880f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:48714/0 (socket says 172.21.15.123:48714) 2024-01-13T11:54:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.763+0000 7f788d4d8700 1 -- 172.21.15.123:0/2633189400 learned_addr learned my addr 172.21.15.123:0/2633189400 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:54:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.763+0000 7f788d4d8700 1 -- 172.21.15.123:0/2633189400 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7884005040 con 0x7f78880601e0 2024-01-13T11:54:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.763+0000 7f788d4d8700 1 --2- 172.21.15.123:0/2633189400 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78880601e0 0x7f78880f6c10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f7884004dd0 tx=0x7f7884009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.763+0000 7f787dffb700 1 -- 172.21.15.123:0/2633189400 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7884018070 con 0x7f78880601e0 2024-01-13T11:54:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.763+0000 7f788f73c700 1 -- 172.21.15.123:0/2633189400 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78880f5230 con 0x7f78880601e0 2024-01-13T11:54:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.763+0000 7f787dffb700 1 -- 172.21.15.123:0/2633189400 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7884013410 con 0x7f78880601e0 2024-01-13T11:54:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.763+0000 7f787dffb700 1 -- 172.21.15.123:0/2633189400 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f788401c3c0 con 0x7f78880601e0 2024-01-13T11:54:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.763+0000 7f788f73c700 1 -- 172.21.15.123:0/2633189400 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78880f56c0 con 0x7f78880601e0 2024-01-13T11:54:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.764+0000 7f787dffb700 1 -- 172.21.15.123:0/2633189400 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f788401c630 con 0x7f78880601e0 2024-01-13T11:54:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.764+0000 7f788f73c700 1 -- 172.21.15.123:0/2633189400 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7888059100 con 0x7f78880601e0 2024-01-13T11:54:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.764+0000 7f787dffb700 1 --2- 172.21.15.123:0/2633189400 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f787402ff60 0x7f7874032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:54:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.764+0000 7f787dffb700 1 -- 172.21.15.123:0/2633189400 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f78840416c0 con 0x7f78880601e0 2024-01-13T11:54:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.765+0000 7f788ccd7700 1 --2- 172.21.15.123:0/2633189400 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f787402ff60 0x7f7874032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:54:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.765+0000 7f788ccd7700 1 --2- 172.21.15.123:0/2633189400 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f787402ff60 0x7f7874032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f78780096a0 tx=0x7f7878006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:54:58.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.769+0000 7f787dffb700 1 -- 172.21.15.123:0/2633189400 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f788401ca30 con 0x7f78880601e0 2024-01-13T11:54:58.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.972+0000 7f788f73c700 1 -- 172.21.15.123:0/2633189400 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78880fbf60 con 0x7f78880601e0 2024-01-13T11:54:58.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.973+0000 7f787dffb700 1 -- 172.21.15.123:0/2633189400 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f788401b370 con 0x7f78880601e0 2024-01-13T11:54:58.961 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:54:58.961 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:54:58.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.975+0000 7f788f73c700 1 -- 172.21.15.123:0/2633189400 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f787402ff60 msgr2=0x7f7874032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:58.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.975+0000 7f788f73c700 1 --2- 172.21.15.123:0/2633189400 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f787402ff60 0x7f7874032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f78780096a0 tx=0x7f7878006b40).stop 2024-01-13T11:54:58.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.975+0000 7f788f73c700 1 -- 172.21.15.123:0/2633189400 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78880601e0 msgr2=0x7f78880f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:54:58.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.975+0000 7f788f73c700 1 --2- 172.21.15.123:0/2633189400 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78880601e0 0x7f78880f6c10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f7884004dd0 tx=0x7f7884009b30).stop 2024-01-13T11:54:58.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.975+0000 7f788f73c700 1 -- 172.21.15.123:0/2633189400 shutdown_connections 2024-01-13T11:54:58.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.975+0000 7f788f73c700 1 --2- 172.21.15.123:0/2633189400 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f787402ff60 0x7f7874032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:58.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.975+0000 7f788f73c700 1 --2- 172.21.15.123:0/2633189400 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78880601e0 0x7f78880f6c10 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:54:58.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.975+0000 7f788f73c700 1 -- 172.21.15.123:0/2633189400 >> 172.21.15.123:0/2633189400 conn(0x7f78880f0b70 msgr2=0x7f78880f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:54:58.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.975+0000 7f788f73c700 1 -- 172.21.15.123:0/2633189400 shutdown_connections 2024-01-13T11:54:58.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:54:58.975+0000 7f788f73c700 1 -- 172.21.15.123:0/2633189400 wait complete. 2024-01-13T11:54:58.964 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:54:59.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:59 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:57.550357+0000 mgr.smithi059.hmyldj (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:59.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:54:59 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:54:58.974296+0000 mon.smithi059 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.123:0/2633189400' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:00.340 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:00.341 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:01.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.460+0000 7fb7c549b700 1 -- 172.21.15.123:0/3016733229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7c00601e0 msgr2=0x7fb7c00f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.460+0000 7fb7c549b700 1 --2- 172.21.15.123:0/3016733229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7c00601e0 0x7fb7c00f8f20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fb7b4009fb0 tx=0x7fb7b4009b30).stop 2024-01-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.460+0000 7fb7c549b700 1 -- 172.21.15.123:0/3016733229 shutdown_connections 2024-01-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.460+0000 7fb7c549b700 1 --2- 172.21.15.123:0/3016733229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7c00601e0 0x7fb7c00f8f20 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.460+0000 7fb7c549b700 1 -- 172.21.15.123:0/3016733229 >> 172.21.15.123:0/3016733229 conn(0x7fb7c00f09a0 msgr2=0x7fb7c00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.460+0000 7fb7c549b700 1 -- 172.21.15.123:0/3016733229 shutdown_connections 2024-01-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.460+0000 7fb7c549b700 1 -- 172.21.15.123:0/3016733229 wait complete. 2024-01-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.460+0000 7fb7c549b700 1 Processor -- start 2024-01-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.460+0000 7fb7c549b700 1 -- start start 2024-01-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.460+0000 7fb7c549b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7c00601e0 0x7fb7c0106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.460+0000 7fb7c549b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c01069f0 con 0x7fb7c00601e0 2024-01-13T11:55:01.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.461+0000 7fb7bffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7c00601e0 0x7fb7c0106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.461+0000 7fb7bffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7c00601e0 0x7fb7c0106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:48720/0 (socket says 172.21.15.123:48720) 2024-01-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.461+0000 7fb7bffff700 1 -- 172.21.15.123:0/347215812 learned_addr learned my addr 172.21.15.123:0/347215812 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.461+0000 7fb7bffff700 1 -- 172.21.15.123:0/347215812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b4005040 con 0x7fb7c00601e0 2024-01-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.461+0000 7fb7bffff700 1 --2- 172.21.15.123:0/347215812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7c00601e0 0x7fb7c0106500 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fb7b4013510 tx=0x7fb7b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.461+0000 7fb7bcff9700 1 -- 172.21.15.123:0/347215812 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7b4009320 con 0x7fb7c00601e0 2024-01-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.461+0000 7fb7c549b700 1 -- 172.21.15.123:0/347215812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c0106bf0 con 0x7fb7c00601e0 2024-01-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.461+0000 7fb7bcff9700 1 -- 172.21.15.123:0/347215812 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7b4013f00 con 0x7fb7c00601e0 2024-01-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.461+0000 7fb7bcff9700 1 -- 172.21.15.123:0/347215812 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7b4014d90 con 0x7fb7c00601e0 2024-01-13T11:55:01.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.461+0000 7fb7c549b700 1 -- 172.21.15.123:0/347215812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c0107080 con 0x7fb7c00601e0 2024-01-13T11:55:01.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.462+0000 7fb7bcff9700 1 -- 172.21.15.123:0/347215812 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb7b40263a0 con 0x7fb7c00601e0 2024-01-13T11:55:01.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.462+0000 7fb7bcff9700 1 --2- 172.21.15.123:0/347215812 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb7a802ff60 0x7fb7a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:01.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.462+0000 7fb7bcff9700 1 -- 172.21.15.123:0/347215812 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb7b40410e0 con 0x7fb7c00601e0 2024-01-13T11:55:01.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.462+0000 7fb7c549b700 1 -- 172.21.15.123:0/347215812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c0045ad0 con 0x7fb7c00601e0 2024-01-13T11:55:01.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.462+0000 7fb7bf7fe700 1 --2- 172.21.15.123:0/347215812 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb7a802ff60 0x7fb7a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:01.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.463+0000 7fb7bf7fe700 1 --2- 172.21.15.123:0/347215812 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb7a802ff60 0x7fb7a8032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fb7b00096a0 tx=0x7fb7b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:01.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.467+0000 7fb7bcff9700 1 -- 172.21.15.123:0/347215812 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb7b4012920 con 0x7fb7c00601e0 2024-01-13T11:55:01.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:01 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:54:59.550698+0000 mgr.smithi059.hmyldj (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:01.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.672+0000 7fb7c549b700 1 -- 172.21.15.123:0/347215812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb7c00fbdc0 con 0x7fb7c00601e0 2024-01-13T11:55:01.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.673+0000 7fb7bcff9700 1 -- 172.21.15.123:0/347215812 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb7b402b020 con 0x7fb7c00601e0 2024-01-13T11:55:01.661 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:01.661 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:01.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.674+0000 7fb7c549b700 1 -- 172.21.15.123:0/347215812 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb7a802ff60 msgr2=0x7fb7a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:01.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.674+0000 7fb7c549b700 1 --2- 172.21.15.123:0/347215812 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb7a802ff60 0x7fb7a8032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fb7b00096a0 tx=0x7fb7b0006b40).stop 2024-01-13T11:55:01.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.675+0000 7fb7c549b700 1 -- 172.21.15.123:0/347215812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7c00601e0 msgr2=0x7fb7c0106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:01.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.675+0000 7fb7c549b700 1 --2- 172.21.15.123:0/347215812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7c00601e0 0x7fb7c0106500 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fb7b4013510 tx=0x7fb7b400b040).stop 2024-01-13T11:55:01.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.675+0000 7fb7c549b700 1 -- 172.21.15.123:0/347215812 shutdown_connections 2024-01-13T11:55:01.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.675+0000 7fb7c549b700 1 --2- 172.21.15.123:0/347215812 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb7a802ff60 0x7fb7a8032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:01.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.675+0000 7fb7c549b700 1 --2- 172.21.15.123:0/347215812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7c00601e0 0x7fb7c0106500 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:01.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.675+0000 7fb7c549b700 1 -- 172.21.15.123:0/347215812 >> 172.21.15.123:0/347215812 conn(0x7fb7c00f09a0 msgr2=0x7fb7c00f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:01.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.675+0000 7fb7c549b700 1 -- 172.21.15.123:0/347215812 shutdown_connections 2024-01-13T11:55:01.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:01.675+0000 7fb7c549b700 1 -- 172.21.15.123:0/347215812 wait complete. 2024-01-13T11:55:01.664 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:02.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:02 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:01.674361+0000 mon.smithi059 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.123:0/347215812' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:03.010 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:03.010 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:03.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:03 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:01.551039+0000 mgr.smithi059.hmyldj (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:04.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.119+0000 7f654f3ab700 1 -- 172.21.15.123:0/2564651879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65480f6510 msgr2=0x7f65480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:04.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.119+0000 7f654f3ab700 1 --2- 172.21.15.123:0/2564651879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65480f6510 0x7f65480f6930 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f6538009960 tx=0x7f6538004d10).stop 2024-01-13T11:55:04.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.119+0000 7f654f3ab700 1 -- 172.21.15.123:0/2564651879 shutdown_connections 2024-01-13T11:55:04.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.119+0000 7f654f3ab700 1 --2- 172.21.15.123:0/2564651879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65480f6510 0x7f65480f6930 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:04.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.119+0000 7f654f3ab700 1 -- 172.21.15.123:0/2564651879 >> 172.21.15.123:0/2564651879 conn(0x7f65480f1c70 msgr2=0x7f65480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:04.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.119+0000 7f654f3ab700 1 -- 172.21.15.123:0/2564651879 shutdown_connections 2024-01-13T11:55:04.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.119+0000 7f654f3ab700 1 -- 172.21.15.123:0/2564651879 wait complete. 2024-01-13T11:55:04.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.120+0000 7f654f3ab700 1 Processor -- start 2024-01-13T11:55:04.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.120+0000 7f654f3ab700 1 -- start start 2024-01-13T11:55:04.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.120+0000 7f654f3ab700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65480f6510 0x7f6548106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:04.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.120+0000 7f654f3ab700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6548106c20 con 0x7f65480f6510 2024-01-13T11:55:04.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.120+0000 7f654d147700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65480f6510 0x7f6548106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:04.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.120+0000 7f654d147700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65480f6510 0x7f6548106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:48728/0 (socket says 172.21.15.123:48728) 2024-01-13T11:55:04.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.120+0000 7f654d147700 1 -- 172.21.15.123:0/1752905947 learned_addr learned my addr 172.21.15.123:0/1752905947 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:04.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.120+0000 7f654d147700 1 -- 172.21.15.123:0/1752905947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6538009640 con 0x7f65480f6510 2024-01-13T11:55:04.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.120+0000 7f654d147700 1 --2- 172.21.15.123:0/1752905947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65480f6510 0x7f6548106730 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f6538004b70 tx=0x7f6538004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:04.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.120+0000 7f6545ffb700 1 -- 172.21.15.123:0/1752905947 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6538018070 con 0x7f65480f6510 2024-01-13T11:55:04.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.121+0000 7f6545ffb700 1 -- 172.21.15.123:0/1752905947 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f653800b870 con 0x7f65480f6510 2024-01-13T11:55:04.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.121+0000 7f654f3ab700 1 -- 172.21.15.123:0/1752905947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6548106e20 con 0x7f65480f6510 2024-01-13T11:55:04.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.121+0000 7f6545ffb700 1 -- 172.21.15.123:0/1752905947 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f653801c560 con 0x7f65480f6510 2024-01-13T11:55:04.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.121+0000 7f654f3ab700 1 -- 172.21.15.123:0/1752905947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65481072b0 con 0x7f65480f6510 2024-01-13T11:55:04.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.121+0000 7f6545ffb700 1 -- 172.21.15.123:0/1752905947 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f653801c6c0 con 0x7f65480f6510 2024-01-13T11:55:04.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.122+0000 7f654f3ab700 1 -- 172.21.15.123:0/1752905947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6548045ad0 con 0x7f65480f6510 2024-01-13T11:55:04.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.122+0000 7f6545ffb700 1 --2- 172.21.15.123:0/1752905947 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f653002ffb0 0x7f6530032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:04.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.122+0000 7f6545ffb700 1 -- 172.21.15.123:0/1752905947 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6538041560 con 0x7f65480f6510 2024-01-13T11:55:04.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.122+0000 7f654c946700 1 --2- 172.21.15.123:0/1752905947 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f653002ffb0 0x7f6530032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:04.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.122+0000 7f654c946700 1 --2- 172.21.15.123:0/1752905947 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f653002ffb0 0x7f6530032470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f653c0096a0 tx=0x7f653c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:04.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.126+0000 7f6545ffb700 1 -- 172.21.15.123:0/1752905947 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6538012560 con 0x7f65480f6510 2024-01-13T11:55:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.334+0000 7f654f3ab700 1 -- 172.21.15.123:0/1752905947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65480fbf60 con 0x7f65480f6510 2024-01-13T11:55:04.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.336+0000 7f6545ffb700 1 -- 172.21.15.123:0/1752905947 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6538012380 con 0x7f65480f6510 2024-01-13T11:55:04.324 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:04.324 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:04.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.337+0000 7f654f3ab700 1 -- 172.21.15.123:0/1752905947 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f653002ffb0 msgr2=0x7f6530032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:04.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.337+0000 7f654f3ab700 1 --2- 172.21.15.123:0/1752905947 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f653002ffb0 0x7f6530032470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f653c0096a0 tx=0x7f653c006b40).stop 2024-01-13T11:55:04.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.337+0000 7f654f3ab700 1 -- 172.21.15.123:0/1752905947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65480f6510 msgr2=0x7f6548106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.337+0000 7f654f3ab700 1 --2- 172.21.15.123:0/1752905947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65480f6510 0x7f6548106730 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f6538004b70 tx=0x7f6538004d10).stop 2024-01-13T11:55:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.337+0000 7f654f3ab700 1 -- 172.21.15.123:0/1752905947 shutdown_connections 2024-01-13T11:55:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.337+0000 7f654f3ab700 1 --2- 172.21.15.123:0/1752905947 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f653002ffb0 0x7f6530032470 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.337+0000 7f654f3ab700 1 --2- 172.21.15.123:0/1752905947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65480f6510 0x7f6548106730 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.337+0000 7f654f3ab700 1 -- 172.21.15.123:0/1752905947 >> 172.21.15.123:0/1752905947 conn(0x7f65480f1c70 msgr2=0x7f65480f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.337+0000 7f654f3ab700 1 -- 172.21.15.123:0/1752905947 shutdown_connections 2024-01-13T11:55:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:04.337+0000 7f654f3ab700 1 -- 172.21.15.123:0/1752905947 wait complete. 2024-01-13T11:55:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:05.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:05 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:03.551266+0000 mgr.smithi059.hmyldj (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:05.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:05 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:04.336334+0000 mon.smithi059 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.123:0/1752905947' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:05.696 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:05.697 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:06.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.838+0000 7f9471aca700 1 -- 172.21.15.123:0/364839512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f946c0f6510 msgr2=0x7f946c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:06.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.838+0000 7f9471aca700 1 --2- 172.21.15.123:0/364839512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f946c0f6510 0x7f946c0f6930 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f9460009fb0 tx=0x7f9460009b30).stop 2024-01-13T11:55:06.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.838+0000 7f9471aca700 1 -- 172.21.15.123:0/364839512 shutdown_connections 2024-01-13T11:55:06.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.838+0000 7f9471aca700 1 --2- 172.21.15.123:0/364839512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f946c0f6510 0x7f946c0f6930 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:06.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.838+0000 7f9471aca700 1 -- 172.21.15.123:0/364839512 >> 172.21.15.123:0/364839512 conn(0x7f946c0f1c70 msgr2=0x7f946c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:06.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.838+0000 7f9471aca700 1 -- 172.21.15.123:0/364839512 shutdown_connections 2024-01-13T11:55:06.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.838+0000 7f9471aca700 1 -- 172.21.15.123:0/364839512 wait complete. 2024-01-13T11:55:06.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.838+0000 7f9471aca700 1 Processor -- start 2024-01-13T11:55:06.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.839+0000 7f9471aca700 1 -- start start 2024-01-13T11:55:06.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.839+0000 7f9471aca700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f946c0f6510 0x7f946c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:06.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.839+0000 7f9471aca700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c106c20 con 0x7f946c0f6510 2024-01-13T11:55:06.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.839+0000 7f946b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f946c0f6510 0x7f946c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:06.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.839+0000 7f946b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f946c0f6510 0x7f946c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:44456/0 (socket says 172.21.15.123:44456) 2024-01-13T11:55:06.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.839+0000 7f946b7fe700 1 -- 172.21.15.123:0/1080244514 learned_addr learned my addr 172.21.15.123:0/1080244514 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:06.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.839+0000 7f946b7fe700 1 -- 172.21.15.123:0/1080244514 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9460005040 con 0x7f946c0f6510 2024-01-13T11:55:06.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.839+0000 7f946b7fe700 1 --2- 172.21.15.123:0/1080244514 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f946c0f6510 0x7f946c106730 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f9460013510 tx=0x7f946000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:06.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.839+0000 7f9453fff700 1 -- 172.21.15.123:0/1080244514 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9460009320 con 0x7f946c0f6510 2024-01-13T11:55:06.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.840+0000 7f9471aca700 1 -- 172.21.15.123:0/1080244514 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946c106e20 con 0x7f946c0f6510 2024-01-13T11:55:06.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.840+0000 7f9453fff700 1 -- 172.21.15.123:0/1080244514 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9460013f00 con 0x7f946c0f6510 2024-01-13T11:55:06.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.840+0000 7f9453fff700 1 -- 172.21.15.123:0/1080244514 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9460014d90 con 0x7f946c0f6510 2024-01-13T11:55:06.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.840+0000 7f9471aca700 1 -- 172.21.15.123:0/1080244514 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f946c1072b0 con 0x7f946c0f6510 2024-01-13T11:55:06.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.840+0000 7f9453fff700 1 -- 172.21.15.123:0/1080244514 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f94600095f0 con 0x7f946c0f6510 2024-01-13T11:55:06.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.840+0000 7f9471aca700 1 -- 172.21.15.123:0/1080244514 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946c045ad0 con 0x7f946c0f6510 2024-01-13T11:55:06.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.840+0000 7f9453fff700 1 --2- 172.21.15.123:0/1080244514 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f945402ff60 0x7f9454032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:06.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.841+0000 7f9453fff700 1 -- 172.21.15.123:0/1080244514 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9460041080 con 0x7f946c0f6510 2024-01-13T11:55:06.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.841+0000 7f946affd700 1 --2- 172.21.15.123:0/1080244514 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f945402ff60 0x7f9454032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:06.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.841+0000 7f946affd700 1 --2- 172.21.15.123:0/1080244514 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f945402ff60 0x7f9454032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f945c0096a0 tx=0x7f945c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:06.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:06.849+0000 7f9453fff700 1 -- 172.21.15.123:0/1080244514 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9460014420 con 0x7f946c0f6510 2024-01-13T11:55:07.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:07.053+0000 7f9471aca700 1 -- 172.21.15.123:0/1080244514 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f946c0fbf60 con 0x7f946c0f6510 2024-01-13T11:55:07.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:07.054+0000 7f9453fff700 1 -- 172.21.15.123:0/1080244514 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9460010230 con 0x7f946c0f6510 2024-01-13T11:55:07.042 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:07.042 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:07.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:07.056+0000 7f9471aca700 1 -- 172.21.15.123:0/1080244514 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f945402ff60 msgr2=0x7f9454032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:07.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:07.056+0000 7f9471aca700 1 --2- 172.21.15.123:0/1080244514 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f945402ff60 0x7f9454032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f945c0096a0 tx=0x7f945c006b40).stop 2024-01-13T11:55:07.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:07.056+0000 7f9471aca700 1 -- 172.21.15.123:0/1080244514 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f946c0f6510 msgr2=0x7f946c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:07.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:07.056+0000 7f9471aca700 1 --2- 172.21.15.123:0/1080244514 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f946c0f6510 0x7f946c106730 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f9460013510 tx=0x7f946000b040).stop 2024-01-13T11:55:07.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:07.056+0000 7f9471aca700 1 -- 172.21.15.123:0/1080244514 shutdown_connections 2024-01-13T11:55:07.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:07.056+0000 7f9471aca700 1 --2- 172.21.15.123:0/1080244514 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f945402ff60 0x7f9454032420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:07.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:07.056+0000 7f9471aca700 1 --2- 172.21.15.123:0/1080244514 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f946c0f6510 0x7f946c106730 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:07.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:07.056+0000 7f9471aca700 1 -- 172.21.15.123:0/1080244514 >> 172.21.15.123:0/1080244514 conn(0x7f946c0f1c70 msgr2=0x7f946c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:07.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:07.056+0000 7f9471aca700 1 -- 172.21.15.123:0/1080244514 shutdown_connections 2024-01-13T11:55:07.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:07.056+0000 7f9471aca700 1 -- 172.21.15.123:0/1080244514 wait complete. 2024-01-13T11:55:07.045 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:07.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:05.551497+0000 mgr.smithi059.hmyldj (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:07.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:07.055227+0000 mon.smithi059 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.123:0/1080244514' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:08.431 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:08.431 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:09.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:09 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:07.551729+0000 mgr.smithi059.hmyldj (mgr.14162) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:09.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.658+0000 7ff4a286b700 1 -- 172.21.15.123:0/2072373546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff49c0f62a0 msgr2=0x7ff49c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:09.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.658+0000 7ff4a286b700 1 --2- 172.21.15.123:0/2072373546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff49c0f62a0 0x7ff49c0f66c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7ff48c009fb0 tx=0x7ff48c009b30).stop 2024-01-13T11:55:09.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.658+0000 7ff4a286b700 1 -- 172.21.15.123:0/2072373546 shutdown_connections 2024-01-13T11:55:09.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.658+0000 7ff4a286b700 1 --2- 172.21.15.123:0/2072373546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff49c0f62a0 0x7ff49c0f66c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:09.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.658+0000 7ff4a286b700 1 -- 172.21.15.123:0/2072373546 >> 172.21.15.123:0/2072373546 conn(0x7ff49c0f1aa0 msgr2=0x7ff49c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:09.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.658+0000 7ff4a286b700 1 -- 172.21.15.123:0/2072373546 shutdown_connections 2024-01-13T11:55:09.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.658+0000 7ff4a286b700 1 -- 172.21.15.123:0/2072373546 wait complete. 2024-01-13T11:55:09.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.658+0000 7ff4a286b700 1 Processor -- start 2024-01-13T11:55:09.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.659+0000 7ff4a286b700 1 -- start start 2024-01-13T11:55:09.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.659+0000 7ff4a286b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff49c0f62a0 0x7ff49c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:09.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.659+0000 7ff4a286b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c069e20 con 0x7ff49c0f62a0 2024-01-13T11:55:09.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.659+0000 7ff4a1869700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff49c0f62a0 0x7ff49c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:09.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.659+0000 7ff4a1869700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff49c0f62a0 0x7ff49c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:44464/0 (socket says 172.21.15.123:44464) 2024-01-13T11:55:09.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.659+0000 7ff4a1869700 1 -- 172.21.15.123:0/3006824097 learned_addr learned my addr 172.21.15.123:0/3006824097 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:09.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.659+0000 7ff4a1869700 1 -- 172.21.15.123:0/3006824097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff48c005040 con 0x7ff49c0f62a0 2024-01-13T11:55:09.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.659+0000 7ff4a1869700 1 --2- 172.21.15.123:0/3006824097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff49c0f62a0 0x7ff49c06b9a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7ff48c013510 tx=0x7ff48c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:09.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.659+0000 7ff4927fc700 1 -- 172.21.15.123:0/3006824097 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff48c009320 con 0x7ff49c0f62a0 2024-01-13T11:55:09.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.659+0000 7ff4927fc700 1 -- 172.21.15.123:0/3006824097 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff48c013f00 con 0x7ff49c0f62a0 2024-01-13T11:55:09.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.659+0000 7ff4a286b700 1 -- 172.21.15.123:0/3006824097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff49c069fc0 con 0x7ff49c0f62a0 2024-01-13T11:55:09.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.659+0000 7ff4927fc700 1 -- 172.21.15.123:0/3006824097 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff48c014dc0 con 0x7ff49c0f62a0 2024-01-13T11:55:09.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.659+0000 7ff4a286b700 1 -- 172.21.15.123:0/3006824097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49c06a450 con 0x7ff49c0f62a0 2024-01-13T11:55:09.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.660+0000 7ff4927fc700 1 -- 172.21.15.123:0/3006824097 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff48c0095f0 con 0x7ff49c0f62a0 2024-01-13T11:55:09.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.660+0000 7ff4a286b700 1 -- 172.21.15.123:0/3006824097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff49c059100 con 0x7ff49c0f62a0 2024-01-13T11:55:09.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.660+0000 7ff4927fc700 1 --2- 172.21.15.123:0/3006824097 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff48802ff60 0x7ff488032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:09.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.660+0000 7ff4927fc700 1 -- 172.21.15.123:0/3006824097 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff48c0410c0 con 0x7ff49c0f62a0 2024-01-13T11:55:09.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.661+0000 7ff4a1068700 1 --2- 172.21.15.123:0/3006824097 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff48802ff60 0x7ff488032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:09.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.661+0000 7ff4a1068700 1 --2- 172.21.15.123:0/3006824097 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff48802ff60 0x7ff488032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7ff4980096a0 tx=0x7ff498006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:09.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.665+0000 7ff4927fc700 1 -- 172.21.15.123:0/3006824097 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff48c01c790 con 0x7ff49c0f62a0 2024-01-13T11:55:09.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.871+0000 7ff4a286b700 1 -- 172.21.15.123:0/3006824097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff49c0fbce0 con 0x7ff49c0f62a0 2024-01-13T11:55:09.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.871+0000 7ff4927fc700 1 -- 172.21.15.123:0/3006824097 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff48c010230 con 0x7ff49c0f62a0 2024-01-13T11:55:09.860 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:09.860 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:09.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.873+0000 7ff4a286b700 1 -- 172.21.15.123:0/3006824097 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff48802ff60 msgr2=0x7ff488032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:09.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.873+0000 7ff4a286b700 1 --2- 172.21.15.123:0/3006824097 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff48802ff60 0x7ff488032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7ff4980096a0 tx=0x7ff498006b40).stop 2024-01-13T11:55:09.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.873+0000 7ff4a286b700 1 -- 172.21.15.123:0/3006824097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff49c0f62a0 msgr2=0x7ff49c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:09.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.873+0000 7ff4a286b700 1 --2- 172.21.15.123:0/3006824097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff49c0f62a0 0x7ff49c06b9a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7ff48c013510 tx=0x7ff48c00b040).stop 2024-01-13T11:55:09.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.873+0000 7ff4a286b700 1 -- 172.21.15.123:0/3006824097 shutdown_connections 2024-01-13T11:55:09.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.873+0000 7ff4a286b700 1 --2- 172.21.15.123:0/3006824097 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff48802ff60 0x7ff488032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:09.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.873+0000 7ff4a286b700 1 --2- 172.21.15.123:0/3006824097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff49c0f62a0 0x7ff49c06b9a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:09.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.873+0000 7ff4a286b700 1 -- 172.21.15.123:0/3006824097 >> 172.21.15.123:0/3006824097 conn(0x7ff49c0f1aa0 msgr2=0x7ff49c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:09.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.873+0000 7ff4a286b700 1 -- 172.21.15.123:0/3006824097 shutdown_connections 2024-01-13T11:55:09.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:09.873+0000 7ff4a286b700 1 -- 172.21.15.123:0/3006824097 wait complete. 2024-01-13T11:55:09.862 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:10.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:10 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:09.872614+0000 mon.smithi059 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.123:0/3006824097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:11.263 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:11.264 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:11.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:11 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:09.551950+0000 mgr.smithi059.hmyldj (mgr.14162) 170 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:12.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.425+0000 7f7ce3a89700 1 -- 172.21.15.123:0/828260114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cdc0f5410 msgr2=0x7f7cdc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:12.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.425+0000 7f7ce3a89700 1 --2- 172.21.15.123:0/828260114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cdc0f5410 0x7f7cdc0f5830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f7ccc009fb0 tx=0x7f7ccc009b30).stop 2024-01-13T11:55:12.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.425+0000 7f7ce3a89700 1 -- 172.21.15.123:0/828260114 shutdown_connections 2024-01-13T11:55:12.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.425+0000 7f7ce3a89700 1 --2- 172.21.15.123:0/828260114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cdc0f5410 0x7f7cdc0f5830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:12.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.425+0000 7f7ce3a89700 1 -- 172.21.15.123:0/828260114 >> 172.21.15.123:0/828260114 conn(0x7f7cdc0f0b70 msgr2=0x7f7cdc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:12.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.425+0000 7f7ce3a89700 1 -- 172.21.15.123:0/828260114 shutdown_connections 2024-01-13T11:55:12.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.425+0000 7f7ce3a89700 1 -- 172.21.15.123:0/828260114 wait complete. 2024-01-13T11:55:12.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.425+0000 7f7ce3a89700 1 Processor -- start 2024-01-13T11:55:12.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.425+0000 7f7ce3a89700 1 -- start start 2024-01-13T11:55:12.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.425+0000 7f7ce3a89700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cdc0f5410 0x7f7cdc1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:12.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.426+0000 7f7ce3a89700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc060280 con 0x7f7cdc0f5410 2024-01-13T11:55:12.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.426+0000 7f7ce1825700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cdc0f5410 0x7f7cdc1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:12.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.426+0000 7f7ce1825700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cdc0f5410 0x7f7cdc1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:44474/0 (socket says 172.21.15.123:44474) 2024-01-13T11:55:12.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.426+0000 7f7ce1825700 1 -- 172.21.15.123:0/1169647652 learned_addr learned my addr 172.21.15.123:0/1169647652 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:12.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.426+0000 7f7ce1825700 1 -- 172.21.15.123:0/1169647652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc005040 con 0x7f7cdc0f5410 2024-01-13T11:55:12.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.426+0000 7f7ce1825700 1 --2- 172.21.15.123:0/1169647652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cdc0f5410 0x7f7cdc1093f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f7ccc013510 tx=0x7f7ccc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:12.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.426+0000 7f7cd27fc700 1 -- 172.21.15.123:0/1169647652 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ccc009320 con 0x7f7cdc0f5410 2024-01-13T11:55:12.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.426+0000 7f7cd27fc700 1 -- 172.21.15.123:0/1169647652 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ccc013f00 con 0x7f7cdc0f5410 2024-01-13T11:55:12.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.426+0000 7f7ce3a89700 1 -- 172.21.15.123:0/1169647652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cdc060480 con 0x7f7cdc0f5410 2024-01-13T11:55:12.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.426+0000 7f7cd27fc700 1 -- 172.21.15.123:0/1169647652 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ccc014d90 con 0x7f7cdc0f5410 2024-01-13T11:55:12.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.426+0000 7f7ce3a89700 1 -- 172.21.15.123:0/1169647652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cdc109b20 con 0x7f7cdc0f5410 2024-01-13T11:55:12.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.427+0000 7f7cd27fc700 1 -- 172.21.15.123:0/1169647652 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ccc0095f0 con 0x7f7cdc0f5410 2024-01-13T11:55:12.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.427+0000 7f7ce3a89700 1 -- 172.21.15.123:0/1169647652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cdc045ad0 con 0x7f7cdc0f5410 2024-01-13T11:55:12.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.427+0000 7f7cd27fc700 1 --2- 172.21.15.123:0/1169647652 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7cc802ff60 0x7f7cc8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:12.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.427+0000 7f7cd27fc700 1 -- 172.21.15.123:0/1169647652 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ccc041080 con 0x7f7cdc0f5410 2024-01-13T11:55:12.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.427+0000 7f7ce1024700 1 --2- 172.21.15.123:0/1169647652 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7cc802ff60 0x7f7cc8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:12.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.428+0000 7f7ce1024700 1 --2- 172.21.15.123:0/1169647652 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7cc802ff60 0x7f7cc8032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f7cd80096a0 tx=0x7f7cd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:12.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.432+0000 7f7cd27fc700 1 -- 172.21.15.123:0/1169647652 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ccc01d7f0 con 0x7f7cdc0f5410 2024-01-13T11:55:12.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.639+0000 7f7ce3a89700 1 -- 172.21.15.123:0/1169647652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7cdc10a340 con 0x7f7cdc0f5410 2024-01-13T11:55:12.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.640+0000 7f7cd27fc700 1 -- 172.21.15.123:0/1169647652 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7ccc010230 con 0x7f7cdc0f5410 2024-01-13T11:55:12.628 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:12.628 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:12.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.641+0000 7f7ce3a89700 1 -- 172.21.15.123:0/1169647652 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7cc802ff60 msgr2=0x7f7cc8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:12.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.641+0000 7f7ce3a89700 1 --2- 172.21.15.123:0/1169647652 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7cc802ff60 0x7f7cc8032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f7cd80096a0 tx=0x7f7cd8006b40).stop 2024-01-13T11:55:12.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.641+0000 7f7ce3a89700 1 -- 172.21.15.123:0/1169647652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cdc0f5410 msgr2=0x7f7cdc1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:12.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.641+0000 7f7ce3a89700 1 --2- 172.21.15.123:0/1169647652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cdc0f5410 0x7f7cdc1093f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f7ccc013510 tx=0x7f7ccc00b040).stop 2024-01-13T11:55:12.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.641+0000 7f7ce3a89700 1 -- 172.21.15.123:0/1169647652 shutdown_connections 2024-01-13T11:55:12.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.641+0000 7f7ce3a89700 1 --2- 172.21.15.123:0/1169647652 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7cc802ff60 0x7f7cc8032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:12.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.641+0000 7f7ce3a89700 1 --2- 172.21.15.123:0/1169647652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cdc0f5410 0x7f7cdc1093f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:12.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.641+0000 7f7ce3a89700 1 -- 172.21.15.123:0/1169647652 >> 172.21.15.123:0/1169647652 conn(0x7f7cdc0f0b70 msgr2=0x7f7cdc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:12.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.641+0000 7f7ce3a89700 1 -- 172.21.15.123:0/1169647652 shutdown_connections 2024-01-13T11:55:12.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:12.641+0000 7f7ce3a89700 1 -- 172.21.15.123:0/1169647652 wait complete. 2024-01-13T11:55:12.631 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:13.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:13 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:11.552265+0000 mgr.smithi059.hmyldj (mgr.14162) 171 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:13.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:13 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:12.445077+0000 mon.smithi059 (mon.0) 242 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' 2024-01-13T11:55:13.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:13 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:12.641011+0000 mon.smithi059 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.123:0/1169647652' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:13.953 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:13.954 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:14.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cephadm 2024-01-13T11:55:12.446687+0000 mgr.smithi059.hmyldj (mgr.14162) 172 : cephadm [INF] Deploying daemon node-exporter.smithi059 on smithi059 2024-01-13T11:55:15.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.104+0000 7faf2a2e0700 1 -- 172.21.15.123:0/1236847108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf240f5410 msgr2=0x7faf240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:15.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.104+0000 7faf2a2e0700 1 --2- 172.21.15.123:0/1236847108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf240f5410 0x7faf240f5830 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7faf14009fb0 tx=0x7faf14009b30).stop 2024-01-13T11:55:15.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.104+0000 7faf2a2e0700 1 -- 172.21.15.123:0/1236847108 shutdown_connections 2024-01-13T11:55:15.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.104+0000 7faf2a2e0700 1 --2- 172.21.15.123:0/1236847108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf240f5410 0x7faf240f5830 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:15.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.104+0000 7faf2a2e0700 1 -- 172.21.15.123:0/1236847108 >> 172.21.15.123:0/1236847108 conn(0x7faf240f0b70 msgr2=0x7faf240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:15.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.104+0000 7faf2a2e0700 1 -- 172.21.15.123:0/1236847108 shutdown_connections 2024-01-13T11:55:15.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.105+0000 7faf2a2e0700 1 -- 172.21.15.123:0/1236847108 wait complete. 2024-01-13T11:55:15.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.105+0000 7faf2a2e0700 1 Processor -- start 2024-01-13T11:55:15.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.105+0000 7faf2a2e0700 1 -- start start 2024-01-13T11:55:15.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.105+0000 7faf2a2e0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf240f5410 0x7faf24109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:15.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.105+0000 7faf2a2e0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf24060220 con 0x7faf240f5410 2024-01-13T11:55:15.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.106+0000 7faf23fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf240f5410 0x7faf24109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:15.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.106+0000 7faf23fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf240f5410 0x7faf24109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:44486/0 (socket says 172.21.15.123:44486) 2024-01-13T11:55:15.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.106+0000 7faf23fff700 1 -- 172.21.15.123:0/1698079189 learned_addr learned my addr 172.21.15.123:0/1698079189 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:15.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.106+0000 7faf23fff700 1 -- 172.21.15.123:0/1698079189 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf14005040 con 0x7faf240f5410 2024-01-13T11:55:15.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.106+0000 7faf23fff700 1 --2- 172.21.15.123:0/1698079189 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf240f5410 0x7faf24109450 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7faf14013510 tx=0x7faf1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:15.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.106+0000 7faf20ff9700 1 -- 172.21.15.123:0/1698079189 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf14009320 con 0x7faf240f5410 2024-01-13T11:55:15.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.106+0000 7faf20ff9700 1 -- 172.21.15.123:0/1698079189 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf14013f00 con 0x7faf240f5410 2024-01-13T11:55:15.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.106+0000 7faf2a2e0700 1 -- 172.21.15.123:0/1698079189 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf24060420 con 0x7faf240f5410 2024-01-13T11:55:15.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.106+0000 7faf20ff9700 1 -- 172.21.15.123:0/1698079189 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf14014d90 con 0x7faf240f5410 2024-01-13T11:55:15.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.106+0000 7faf2a2e0700 1 -- 172.21.15.123:0/1698079189 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf24109b00 con 0x7faf240f5410 2024-01-13T11:55:15.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.107+0000 7faf20ff9700 1 -- 172.21.15.123:0/1698079189 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf14014420 con 0x7faf240f5410 2024-01-13T11:55:15.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.107+0000 7faf2a2e0700 1 -- 172.21.15.123:0/1698079189 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf24059100 con 0x7faf240f5410 2024-01-13T11:55:15.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.107+0000 7faf20ff9700 1 --2- 172.21.15.123:0/1698079189 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7faf1002ff60 0x7faf10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:15.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.107+0000 7faf20ff9700 1 -- 172.21.15.123:0/1698079189 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf14041070 con 0x7faf240f5410 2024-01-13T11:55:15.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.107+0000 7faf237fe700 1 --2- 172.21.15.123:0/1698079189 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7faf1002ff60 0x7faf10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:15.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.108+0000 7faf237fe700 1 --2- 172.21.15.123:0/1698079189 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7faf1002ff60 0x7faf10032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7faf0c0096a0 tx=0x7faf0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:15.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.112+0000 7faf20ff9700 1 -- 172.21.15.123:0/1698079189 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf140095f0 con 0x7faf240f5410 2024-01-13T11:55:15.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.316+0000 7faf2a2e0700 1 -- 172.21.15.123:0/1698079189 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf24045ad0 con 0x7faf240f5410 2024-01-13T11:55:15.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.317+0000 7faf20ff9700 1 -- 172.21.15.123:0/1698079189 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faf1401db70 con 0x7faf240f5410 2024-01-13T11:55:15.305 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:15.305 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:15.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.318+0000 7faf2a2e0700 1 -- 172.21.15.123:0/1698079189 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7faf1002ff60 msgr2=0x7faf10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:15.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.318+0000 7faf2a2e0700 1 --2- 172.21.15.123:0/1698079189 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7faf1002ff60 0x7faf10032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7faf0c0096a0 tx=0x7faf0c006b40).stop 2024-01-13T11:55:15.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.318+0000 7faf2a2e0700 1 -- 172.21.15.123:0/1698079189 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf240f5410 msgr2=0x7faf24109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:15.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.318+0000 7faf2a2e0700 1 --2- 172.21.15.123:0/1698079189 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf240f5410 0x7faf24109450 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7faf14013510 tx=0x7faf1400b040).stop 2024-01-13T11:55:15.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.319+0000 7faf2a2e0700 1 -- 172.21.15.123:0/1698079189 shutdown_connections 2024-01-13T11:55:15.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.319+0000 7faf2a2e0700 1 --2- 172.21.15.123:0/1698079189 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7faf1002ff60 0x7faf10032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:15.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.319+0000 7faf2a2e0700 1 --2- 172.21.15.123:0/1698079189 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf240f5410 0x7faf24109450 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:15.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.319+0000 7faf2a2e0700 1 -- 172.21.15.123:0/1698079189 >> 172.21.15.123:0/1698079189 conn(0x7faf240f0b70 msgr2=0x7faf240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:15.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.319+0000 7faf2a2e0700 1 -- 172.21.15.123:0/1698079189 shutdown_connections 2024-01-13T11:55:15.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:15.319+0000 7faf2a2e0700 1 -- 172.21.15.123:0/1698079189 wait complete. 2024-01-13T11:55:15.308 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:15.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:15 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:13.552650+0000 mgr.smithi059.hmyldj (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:16.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:16 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:15.318370+0000 mon.smithi059 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.123:0/1698079189' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:16.672 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:16.672 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:17.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:17 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:15.552969+0000 mgr.smithi059.hmyldj (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:17.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.967+0000 7fba3d25e700 1 -- 172.21.15.123:0/1215603799 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba380601e0 msgr2=0x7fba380f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:17.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.967+0000 7fba3d25e700 1 --2- 172.21.15.123:0/1215603799 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba380601e0 0x7fba380f8f20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fba28009fb0 tx=0x7fba28009b30).stop 2024-01-13T11:55:17.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.967+0000 7fba3d25e700 1 -- 172.21.15.123:0/1215603799 shutdown_connections 2024-01-13T11:55:17.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.967+0000 7fba3d25e700 1 --2- 172.21.15.123:0/1215603799 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba380601e0 0x7fba380f8f20 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:17.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.967+0000 7fba3d25e700 1 -- 172.21.15.123:0/1215603799 >> 172.21.15.123:0/1215603799 conn(0x7fba380f09a0 msgr2=0x7fba380f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:17.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.967+0000 7fba3d25e700 1 -- 172.21.15.123:0/1215603799 shutdown_connections 2024-01-13T11:55:17.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.967+0000 7fba3d25e700 1 -- 172.21.15.123:0/1215603799 wait complete. 2024-01-13T11:55:17.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.968+0000 7fba3d25e700 1 Processor -- start 2024-01-13T11:55:17.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.968+0000 7fba3d25e700 1 -- start start 2024-01-13T11:55:17.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.968+0000 7fba3d25e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba380601e0 0x7fba38106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:17.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.968+0000 7fba3d25e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba381069f0 con 0x7fba380601e0 2024-01-13T11:55:17.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.969+0000 7fba37fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba380601e0 0x7fba38106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:17.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.969+0000 7fba37fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba380601e0 0x7fba38106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:56728/0 (socket says 172.21.15.123:56728) 2024-01-13T11:55:17.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.969+0000 7fba37fff700 1 -- 172.21.15.123:0/1441540238 learned_addr learned my addr 172.21.15.123:0/1441540238 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:17.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.969+0000 7fba37fff700 1 -- 172.21.15.123:0/1441540238 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba28005040 con 0x7fba380601e0 2024-01-13T11:55:17.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.969+0000 7fba37fff700 1 --2- 172.21.15.123:0/1441540238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba380601e0 0x7fba38106500 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fba28013510 tx=0x7fba2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:17.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.969+0000 7fba34ff9700 1 -- 172.21.15.123:0/1441540238 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba28009320 con 0x7fba380601e0 2024-01-13T11:55:17.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.969+0000 7fba3d25e700 1 -- 172.21.15.123:0/1441540238 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba38106bf0 con 0x7fba380601e0 2024-01-13T11:55:17.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.969+0000 7fba34ff9700 1 -- 172.21.15.123:0/1441540238 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba28013f00 con 0x7fba380601e0 2024-01-13T11:55:17.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.969+0000 7fba34ff9700 1 -- 172.21.15.123:0/1441540238 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba28014d90 con 0x7fba380601e0 2024-01-13T11:55:17.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.969+0000 7fba3d25e700 1 -- 172.21.15.123:0/1441540238 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba38107080 con 0x7fba380601e0 2024-01-13T11:55:17.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.970+0000 7fba34ff9700 1 -- 172.21.15.123:0/1441540238 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba280263a0 con 0x7fba380601e0 2024-01-13T11:55:17.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.970+0000 7fba34ff9700 1 --2- 172.21.15.123:0/1441540238 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fba2002ff60 0x7fba20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:17.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.970+0000 7fba34ff9700 1 -- 172.21.15.123:0/1441540238 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba280410a0 con 0x7fba380601e0 2024-01-13T11:55:17.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.970+0000 7fba3d25e700 1 -- 172.21.15.123:0/1441540238 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba38045ad0 con 0x7fba380601e0 2024-01-13T11:55:17.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.970+0000 7fba377fe700 1 --2- 172.21.15.123:0/1441540238 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fba2002ff60 0x7fba20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:17.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.971+0000 7fba377fe700 1 --2- 172.21.15.123:0/1441540238 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fba2002ff60 0x7fba20032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fba2c0096a0 tx=0x7fba2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:17.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:17.975+0000 7fba34ff9700 1 -- 172.21.15.123:0/1441540238 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba28012920 con 0x7fba380601e0 2024-01-13T11:55:18.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:18.178+0000 7fba3d25e700 1 -- 172.21.15.123:0/1441540238 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba38107da0 con 0x7fba380601e0 2024-01-13T11:55:18.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:18.179+0000 7fba34ff9700 1 -- 172.21.15.123:0/1441540238 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fba2802b020 con 0x7fba380601e0 2024-01-13T11:55:18.167 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:18.167 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:18.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:18.180+0000 7fba3d25e700 1 -- 172.21.15.123:0/1441540238 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fba2002ff60 msgr2=0x7fba20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:18.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:18.180+0000 7fba3d25e700 1 --2- 172.21.15.123:0/1441540238 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fba2002ff60 0x7fba20032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fba2c0096a0 tx=0x7fba2c006b40).stop 2024-01-13T11:55:18.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:18.180+0000 7fba3d25e700 1 -- 172.21.15.123:0/1441540238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba380601e0 msgr2=0x7fba38106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:18.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:18.180+0000 7fba3d25e700 1 --2- 172.21.15.123:0/1441540238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba380601e0 0x7fba38106500 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fba28013510 tx=0x7fba2800b040).stop 2024-01-13T11:55:18.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:18.180+0000 7fba3d25e700 1 -- 172.21.15.123:0/1441540238 shutdown_connections 2024-01-13T11:55:18.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:18.180+0000 7fba3d25e700 1 --2- 172.21.15.123:0/1441540238 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fba2002ff60 0x7fba20032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:18.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:18.180+0000 7fba3d25e700 1 --2- 172.21.15.123:0/1441540238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba380601e0 0x7fba38106500 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:18.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:18.180+0000 7fba3d25e700 1 -- 172.21.15.123:0/1441540238 >> 172.21.15.123:0/1441540238 conn(0x7fba380f09a0 msgr2=0x7fba380f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:18.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:18.180+0000 7fba3d25e700 1 -- 172.21.15.123:0/1441540238 shutdown_connections 2024-01-13T11:55:18.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:18.180+0000 7fba3d25e700 1 -- 172.21.15.123:0/1441540238 wait complete. 2024-01-13T11:55:18.169 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:18.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:18 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:18.180216+0000 mon.smithi059 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.123:0/1441540238' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:19.567 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:19.567 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:19.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:19 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:17.553257+0000 mgr.smithi059.hmyldj (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:20.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.741+0000 7f21b34ef700 1 -- 172.21.15.123:0/1278122154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21ac0f6430 msgr2=0x7f21ac0f6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:20.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.741+0000 7f21b34ef700 1 --2- 172.21.15.123:0/1278122154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21ac0f6430 0x7f21ac0f6850 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f21a0009fb0 tx=0x7f21a0009b30).stop 2024-01-13T11:55:20.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.741+0000 7f21b34ef700 1 -- 172.21.15.123:0/1278122154 shutdown_connections 2024-01-13T11:55:20.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.741+0000 7f21b34ef700 1 --2- 172.21.15.123:0/1278122154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21ac0f6430 0x7f21ac0f6850 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:20.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.741+0000 7f21b34ef700 1 -- 172.21.15.123:0/1278122154 >> 172.21.15.123:0/1278122154 conn(0x7f21ac0f1b90 msgr2=0x7f21ac0f3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:20.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.741+0000 7f21b34ef700 1 -- 172.21.15.123:0/1278122154 shutdown_connections 2024-01-13T11:55:20.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.741+0000 7f21b34ef700 1 -- 172.21.15.123:0/1278122154 wait complete. 2024-01-13T11:55:20.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.742+0000 7f21b34ef700 1 Processor -- start 2024-01-13T11:55:20.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.742+0000 7f21b34ef700 1 -- start start 2024-01-13T11:55:20.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.742+0000 7f21b34ef700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21ac0f6430 0x7f21ac06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:20.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.742+0000 7f21b34ef700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac069e20 con 0x7f21ac0f6430 2024-01-13T11:55:20.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.743+0000 7f21b128b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21ac0f6430 0x7f21ac06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:20.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.743+0000 7f21b128b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21ac0f6430 0x7f21ac06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:56740/0 (socket says 172.21.15.123:56740) 2024-01-13T11:55:20.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.743+0000 7f21b128b700 1 -- 172.21.15.123:0/857501757 learned_addr learned my addr 172.21.15.123:0/857501757 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:20.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.743+0000 7f21b128b700 1 -- 172.21.15.123:0/857501757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a0005040 con 0x7f21ac0f6430 2024-01-13T11:55:20.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.743+0000 7f21b128b700 1 --2- 172.21.15.123:0/857501757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21ac0f6430 0x7f21ac06b9a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f21a0013510 tx=0x7f21a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:20.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.743+0000 7f219dffb700 1 -- 172.21.15.123:0/857501757 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21a0009320 con 0x7f21ac0f6430 2024-01-13T11:55:20.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.743+0000 7f219dffb700 1 -- 172.21.15.123:0/857501757 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21a0013f00 con 0x7f21ac0f6430 2024-01-13T11:55:20.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.743+0000 7f21b34ef700 1 -- 172.21.15.123:0/857501757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21ac069fc0 con 0x7f21ac0f6430 2024-01-13T11:55:20.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.743+0000 7f219dffb700 1 -- 172.21.15.123:0/857501757 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21a0014d90 con 0x7f21ac0f6430 2024-01-13T11:55:20.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.743+0000 7f21b34ef700 1 -- 172.21.15.123:0/857501757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21ac06a450 con 0x7f21ac0f6430 2024-01-13T11:55:20.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.744+0000 7f21b34ef700 1 -- 172.21.15.123:0/857501757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21ac045ad0 con 0x7f21ac0f6430 2024-01-13T11:55:20.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.744+0000 7f219dffb700 1 -- 172.21.15.123:0/857501757 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f21a00095f0 con 0x7f21ac0f6430 2024-01-13T11:55:20.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.744+0000 7f219dffb700 1 --2- 172.21.15.123:0/857501757 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f219802ff60 0x7f2198032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:20.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.744+0000 7f219dffb700 1 -- 172.21.15.123:0/857501757 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f21a00410c0 con 0x7f21ac0f6430 2024-01-13T11:55:20.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.745+0000 7f21b0a8a700 1 --2- 172.21.15.123:0/857501757 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f219802ff60 0x7f2198032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:20.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.745+0000 7f21b0a8a700 1 --2- 172.21.15.123:0/857501757 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f219802ff60 0x7f2198032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f21a80096a0 tx=0x7f21a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:20.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.750+0000 7f219dffb700 1 -- 172.21.15.123:0/857501757 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f21a0014420 con 0x7f21ac0f6430 2024-01-13T11:55:20.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.954+0000 7f21b34ef700 1 -- 172.21.15.123:0/857501757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21ac05ea50 con 0x7f21ac0f6430 2024-01-13T11:55:20.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.954+0000 7f219dffb700 1 -- 172.21.15.123:0/857501757 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f21a001d7f0 con 0x7f21ac0f6430 2024-01-13T11:55:20.943 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:20.943 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:20.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.956+0000 7f21b34ef700 1 -- 172.21.15.123:0/857501757 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f219802ff60 msgr2=0x7f2198032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:20.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.956+0000 7f21b34ef700 1 --2- 172.21.15.123:0/857501757 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f219802ff60 0x7f2198032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f21a80096a0 tx=0x7f21a8006b40).stop 2024-01-13T11:55:20.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.956+0000 7f21b34ef700 1 -- 172.21.15.123:0/857501757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21ac0f6430 msgr2=0x7f21ac06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:20.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.956+0000 7f21b34ef700 1 --2- 172.21.15.123:0/857501757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21ac0f6430 0x7f21ac06b9a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f21a0013510 tx=0x7f21a000b040).stop 2024-01-13T11:55:20.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.956+0000 7f21b34ef700 1 -- 172.21.15.123:0/857501757 shutdown_connections 2024-01-13T11:55:20.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.956+0000 7f21b34ef700 1 --2- 172.21.15.123:0/857501757 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f219802ff60 0x7f2198032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:20.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.956+0000 7f21b34ef700 1 --2- 172.21.15.123:0/857501757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21ac0f6430 0x7f21ac06b9a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:20.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.956+0000 7f21b34ef700 1 -- 172.21.15.123:0/857501757 >> 172.21.15.123:0/857501757 conn(0x7f21ac0f1b90 msgr2=0x7f21ac0f2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:20.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.956+0000 7f21b34ef700 1 -- 172.21.15.123:0/857501757 shutdown_connections 2024-01-13T11:55:20.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:20.956+0000 7f21b34ef700 1 -- 172.21.15.123:0/857501757 wait complete. 2024-01-13T11:55:20.945 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:21.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:21 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:19.553498+0000 mgr.smithi059.hmyldj (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:21.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:21 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:20.955790+0000 mon.smithi059 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.123:0/857501757' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:22.320 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:22.321 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:22.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:22 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:21.553859+0000 mgr.smithi059.hmyldj (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:23.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.499+0000 7fdcf0ce6700 1 -- 172.21.15.123:0/4000818228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcec060200 msgr2=0x7fdcec0f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:23.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.499+0000 7fdcf0ce6700 1 --2- 172.21.15.123:0/4000818228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcec060200 0x7fdcec0f9170 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fdcdc009fb0 tx=0x7fdcdc009b30).stop 2024-01-13T11:55:23.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.499+0000 7fdcf0ce6700 1 -- 172.21.15.123:0/4000818228 shutdown_connections 2024-01-13T11:55:23.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.499+0000 7fdcf0ce6700 1 --2- 172.21.15.123:0/4000818228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcec060200 0x7fdcec0f9170 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:23.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.499+0000 7fdcf0ce6700 1 -- 172.21.15.123:0/4000818228 >> 172.21.15.123:0/4000818228 conn(0x7fdcec0f0b70 msgr2=0x7fdcec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:23.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.499+0000 7fdcf0ce6700 1 -- 172.21.15.123:0/4000818228 shutdown_connections 2024-01-13T11:55:23.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.499+0000 7fdcf0ce6700 1 -- 172.21.15.123:0/4000818228 wait complete. 2024-01-13T11:55:23.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.500+0000 7fdcf0ce6700 1 Processor -- start 2024-01-13T11:55:23.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.500+0000 7fdcf0ce6700 1 -- start start 2024-01-13T11:55:23.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.500+0000 7fdcf0ce6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcec060200 0x7fdcec1066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:23.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.500+0000 7fdcf0ce6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec106bb0 con 0x7fdcec060200 2024-01-13T11:55:23.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.501+0000 7fdcea59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcec060200 0x7fdcec1066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:23.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.501+0000 7fdcea59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcec060200 0x7fdcec1066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:56756/0 (socket says 172.21.15.123:56756) 2024-01-13T11:55:23.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.501+0000 7fdcea59c700 1 -- 172.21.15.123:0/262028987 learned_addr learned my addr 172.21.15.123:0/262028987 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:23.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.501+0000 7fdcea59c700 1 -- 172.21.15.123:0/262028987 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcdc005040 con 0x7fdcec060200 2024-01-13T11:55:23.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.501+0000 7fdcea59c700 1 --2- 172.21.15.123:0/262028987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcec060200 0x7fdcec1066c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fdcdc013510 tx=0x7fdcdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:23.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.501+0000 7fdce2ffd700 1 -- 172.21.15.123:0/262028987 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcdc009320 con 0x7fdcec060200 2024-01-13T11:55:23.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.501+0000 7fdcf0ce6700 1 -- 172.21.15.123:0/262028987 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcec106db0 con 0x7fdcec060200 2024-01-13T11:55:23.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.501+0000 7fdce2ffd700 1 -- 172.21.15.123:0/262028987 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdcdc013f00 con 0x7fdcec060200 2024-01-13T11:55:23.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.501+0000 7fdce2ffd700 1 -- 172.21.15.123:0/262028987 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcdc014dc0 con 0x7fdcec060200 2024-01-13T11:55:23.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.501+0000 7fdcf0ce6700 1 -- 172.21.15.123:0/262028987 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcec107240 con 0x7fdcec060200 2024-01-13T11:55:23.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.502+0000 7fdce2ffd700 1 -- 172.21.15.123:0/262028987 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdcdc014420 con 0x7fdcec060200 2024-01-13T11:55:23.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.502+0000 7fdcf0ce6700 1 -- 172.21.15.123:0/262028987 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcec045ad0 con 0x7fdcec060200 2024-01-13T11:55:23.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.502+0000 7fdce2ffd700 1 --2- 172.21.15.123:0/262028987 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fdcd802ff60 0x7fdcd8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:23.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.502+0000 7fdce2ffd700 1 -- 172.21.15.123:0/262028987 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdcdc041030 con 0x7fdcec060200 2024-01-13T11:55:23.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.503+0000 7fdce9d9b700 1 --2- 172.21.15.123:0/262028987 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fdcd802ff60 0x7fdcd8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:23.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.503+0000 7fdce9d9b700 1 --2- 172.21.15.123:0/262028987 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fdcd802ff60 0x7fdcd8032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fdcd40096a0 tx=0x7fdcd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:23.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.507+0000 7fdce2ffd700 1 -- 172.21.15.123:0/262028987 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdcdc0095f0 con 0x7fdcec060200 2024-01-13T11:55:23.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.713+0000 7fdcf0ce6700 1 -- 172.21.15.123:0/262028987 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdcec1080f0 con 0x7fdcec060200 2024-01-13T11:55:23.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.713+0000 7fdce2ffd700 1 -- 172.21.15.123:0/262028987 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdcdc01cb10 con 0x7fdcec060200 2024-01-13T11:55:23.701 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:23.701 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:23.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.715+0000 7fdcf0ce6700 1 -- 172.21.15.123:0/262028987 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fdcd802ff60 msgr2=0x7fdcd8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:23.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.715+0000 7fdcf0ce6700 1 --2- 172.21.15.123:0/262028987 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fdcd802ff60 0x7fdcd8032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fdcd40096a0 tx=0x7fdcd4006b40).stop 2024-01-13T11:55:23.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.715+0000 7fdcf0ce6700 1 -- 172.21.15.123:0/262028987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcec060200 msgr2=0x7fdcec1066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:23.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.715+0000 7fdcf0ce6700 1 --2- 172.21.15.123:0/262028987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcec060200 0x7fdcec1066c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fdcdc013510 tx=0x7fdcdc00b040).stop 2024-01-13T11:55:23.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.715+0000 7fdcf0ce6700 1 -- 172.21.15.123:0/262028987 shutdown_connections 2024-01-13T11:55:23.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.715+0000 7fdcf0ce6700 1 --2- 172.21.15.123:0/262028987 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fdcd802ff60 0x7fdcd8032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:23.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.715+0000 7fdcf0ce6700 1 --2- 172.21.15.123:0/262028987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcec060200 0x7fdcec1066c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:23.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.715+0000 7fdcf0ce6700 1 -- 172.21.15.123:0/262028987 >> 172.21.15.123:0/262028987 conn(0x7fdcec0f0b70 msgr2=0x7fdcec0f16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:23.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.715+0000 7fdcf0ce6700 1 -- 172.21.15.123:0/262028987 shutdown_connections 2024-01-13T11:55:23.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:23.715+0000 7fdcf0ce6700 1 -- 172.21.15.123:0/262028987 wait complete. 2024-01-13T11:55:23.703 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:24.886 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:24 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:23.554156+0000 mgr.smithi059.hmyldj (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:24.887 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:24 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:23.713691+0000 mon.smithi059 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.123:0/262028987' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:25.098 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:25.099 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:26.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.292+0000 7f100ed81700 1 -- 172.21.15.123:0/1328687686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10080f6510 msgr2=0x7f10080f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:26.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.292+0000 7f100ed81700 1 --2- 172.21.15.123:0/1328687686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10080f6510 0x7f10080f6930 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f0ffc009fb0 tx=0x7f0ffc009b30).stop 2024-01-13T11:55:26.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.292+0000 7f100ed81700 1 -- 172.21.15.123:0/1328687686 shutdown_connections 2024-01-13T11:55:26.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.292+0000 7f100ed81700 1 --2- 172.21.15.123:0/1328687686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10080f6510 0x7f10080f6930 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:26.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.292+0000 7f100ed81700 1 -- 172.21.15.123:0/1328687686 >> 172.21.15.123:0/1328687686 conn(0x7f10080f1c70 msgr2=0x7f10080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:26.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.292+0000 7f100ed81700 1 -- 172.21.15.123:0/1328687686 shutdown_connections 2024-01-13T11:55:26.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.292+0000 7f100ed81700 1 -- 172.21.15.123:0/1328687686 wait complete. 2024-01-13T11:55:26.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.292+0000 7f100ed81700 1 Processor -- start 2024-01-13T11:55:26.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.292+0000 7f100ed81700 1 -- start start 2024-01-13T11:55:26.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.293+0000 7f100ed81700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10080f6510 0x7f1008106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:26.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.293+0000 7f100ed81700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1008106c20 con 0x7f10080f6510 2024-01-13T11:55:26.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.293+0000 7f100cb1d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10080f6510 0x7f1008106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:26.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.293+0000 7f100cb1d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10080f6510 0x7f1008106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:56776/0 (socket says 172.21.15.123:56776) 2024-01-13T11:55:26.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.293+0000 7f100cb1d700 1 -- 172.21.15.123:0/1214171147 learned_addr learned my addr 172.21.15.123:0/1214171147 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:26.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.293+0000 7f100cb1d700 1 -- 172.21.15.123:0/1214171147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ffc005040 con 0x7f10080f6510 2024-01-13T11:55:26.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.293+0000 7f100cb1d700 1 --2- 172.21.15.123:0/1214171147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10080f6510 0x7f1008106730 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f0ffc013510 tx=0x7f0ffc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:26.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.293+0000 7f10057fa700 1 -- 172.21.15.123:0/1214171147 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ffc009320 con 0x7f10080f6510 2024-01-13T11:55:26.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.294+0000 7f10057fa700 1 -- 172.21.15.123:0/1214171147 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ffc013f00 con 0x7f10080f6510 2024-01-13T11:55:26.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.294+0000 7f10057fa700 1 -- 172.21.15.123:0/1214171147 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ffc014d90 con 0x7f10080f6510 2024-01-13T11:55:26.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.294+0000 7f100ed81700 1 -- 172.21.15.123:0/1214171147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1008106e20 con 0x7f10080f6510 2024-01-13T11:55:26.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.294+0000 7f100ed81700 1 -- 172.21.15.123:0/1214171147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10081072b0 con 0x7f10080f6510 2024-01-13T11:55:26.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.294+0000 7f100ed81700 1 -- 172.21.15.123:0/1214171147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1008045ad0 con 0x7f10080f6510 2024-01-13T11:55:26.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.295+0000 7f10057fa700 1 -- 172.21.15.123:0/1214171147 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ffc0095f0 con 0x7f10080f6510 2024-01-13T11:55:26.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.295+0000 7f10057fa700 1 --2- 172.21.15.123:0/1214171147 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0ff802ffb0 0x7f0ff8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:26.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.295+0000 7f10057fa700 1 -- 172.21.15.123:0/1214171147 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ffc040fb0 con 0x7f10080f6510 2024-01-13T11:55:26.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.295+0000 7f1007fff700 1 --2- 172.21.15.123:0/1214171147 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0ff802ffb0 0x7f0ff8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:26.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.295+0000 7f1007fff700 1 --2- 172.21.15.123:0/1214171147 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0ff802ffb0 0x7f0ff8032470 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f0ff40096a0 tx=0x7f0ff4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:26.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.299+0000 7f10057fa700 1 -- 172.21.15.123:0/1214171147 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ffc06d2b0 con 0x7f10080f6510 2024-01-13T11:55:26.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.506+0000 7f100ed81700 1 -- 172.21.15.123:0/1214171147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10080fbf60 con 0x7f10080f6510 2024-01-13T11:55:26.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.507+0000 7f10057fa700 1 -- 172.21.15.123:0/1214171147 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ffc014420 con 0x7f10080f6510 2024-01-13T11:55:26.494 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:26.494 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:26.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.508+0000 7f100ed81700 1 -- 172.21.15.123:0/1214171147 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0ff802ffb0 msgr2=0x7f0ff8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:26.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.508+0000 7f100ed81700 1 --2- 172.21.15.123:0/1214171147 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0ff802ffb0 0x7f0ff8032470 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f0ff40096a0 tx=0x7f0ff4006b40).stop 2024-01-13T11:55:26.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.508+0000 7f100ed81700 1 -- 172.21.15.123:0/1214171147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10080f6510 msgr2=0x7f1008106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:26.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.508+0000 7f100ed81700 1 --2- 172.21.15.123:0/1214171147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10080f6510 0x7f1008106730 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f0ffc013510 tx=0x7f0ffc00b040).stop 2024-01-13T11:55:26.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.508+0000 7f100ed81700 1 -- 172.21.15.123:0/1214171147 shutdown_connections 2024-01-13T11:55:26.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.508+0000 7f100ed81700 1 --2- 172.21.15.123:0/1214171147 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0ff802ffb0 0x7f0ff8032470 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:26.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.508+0000 7f100ed81700 1 --2- 172.21.15.123:0/1214171147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10080f6510 0x7f1008106730 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:26.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.508+0000 7f100ed81700 1 -- 172.21.15.123:0/1214171147 >> 172.21.15.123:0/1214171147 conn(0x7f10080f1c70 msgr2=0x7f10080f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:26.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.508+0000 7f100ed81700 1 -- 172.21.15.123:0/1214171147 shutdown_connections 2024-01-13T11:55:26.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:26.508+0000 7f100ed81700 1 -- 172.21.15.123:0/1214171147 wait complete. 2024-01-13T11:55:26.496 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:27.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:27 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:25.554530+0000 mgr.smithi059.hmyldj (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:27.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:27 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:26.507121+0000 mon.smithi059 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.123:0/1214171147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:27.876 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:27.877 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:29.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.022+0000 7f2d6aad2700 1 -- 172.21.15.123:0/1461557779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d640f5410 msgr2=0x7f2d640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:29.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.022+0000 7f2d6aad2700 1 --2- 172.21.15.123:0/1461557779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d640f5410 0x7f2d640f5830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f2d58009fb0 tx=0x7f2d58009b30).stop 2024-01-13T11:55:29.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.022+0000 7f2d6aad2700 1 -- 172.21.15.123:0/1461557779 shutdown_connections 2024-01-13T11:55:29.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.022+0000 7f2d6aad2700 1 --2- 172.21.15.123:0/1461557779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d640f5410 0x7f2d640f5830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:29.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.022+0000 7f2d6aad2700 1 -- 172.21.15.123:0/1461557779 >> 172.21.15.123:0/1461557779 conn(0x7f2d640f0b70 msgr2=0x7f2d640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:29.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.022+0000 7f2d6aad2700 1 -- 172.21.15.123:0/1461557779 shutdown_connections 2024-01-13T11:55:29.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.022+0000 7f2d6aad2700 1 -- 172.21.15.123:0/1461557779 wait complete. 2024-01-13T11:55:29.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.023+0000 7f2d6aad2700 1 Processor -- start 2024-01-13T11:55:29.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.023+0000 7f2d6aad2700 1 -- start start 2024-01-13T11:55:29.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.023+0000 7f2d6aad2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d640f5410 0x7f2d641093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:29.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.023+0000 7f2d6aad2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d64060280 con 0x7f2d640f5410 2024-01-13T11:55:29.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.023+0000 7f2d6886e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d640f5410 0x7f2d641093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:29.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.023+0000 7f2d6886e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d640f5410 0x7f2d641093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:39338/0 (socket says 172.21.15.123:39338) 2024-01-13T11:55:29.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.023+0000 7f2d6886e700 1 -- 172.21.15.123:0/4143521728 learned_addr learned my addr 172.21.15.123:0/4143521728 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:29.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.024+0000 7f2d6886e700 1 -- 172.21.15.123:0/4143521728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d58005040 con 0x7f2d640f5410 2024-01-13T11:55:29.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.024+0000 7f2d6886e700 1 --2- 172.21.15.123:0/4143521728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d640f5410 0x7f2d641093f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f2d58013510 tx=0x7f2d5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:29.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.024+0000 7f2d617fa700 1 -- 172.21.15.123:0/4143521728 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d58009320 con 0x7f2d640f5410 2024-01-13T11:55:29.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.024+0000 7f2d6aad2700 1 -- 172.21.15.123:0/4143521728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d64060480 con 0x7f2d640f5410 2024-01-13T11:55:29.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.024+0000 7f2d617fa700 1 -- 172.21.15.123:0/4143521728 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d58013f00 con 0x7f2d640f5410 2024-01-13T11:55:29.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.024+0000 7f2d6aad2700 1 -- 172.21.15.123:0/4143521728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d64109b20 con 0x7f2d640f5410 2024-01-13T11:55:29.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.024+0000 7f2d617fa700 1 -- 172.21.15.123:0/4143521728 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d58014d90 con 0x7f2d640f5410 2024-01-13T11:55:29.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.025+0000 7f2d617fa700 1 -- 172.21.15.123:0/4143521728 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d58014420 con 0x7f2d640f5410 2024-01-13T11:55:29.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.025+0000 7f2d6aad2700 1 -- 172.21.15.123:0/4143521728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d64045ad0 con 0x7f2d640f5410 2024-01-13T11:55:29.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.025+0000 7f2d617fa700 1 --2- 172.21.15.123:0/4143521728 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f2d5402ff60 0x7f2d54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:29.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.025+0000 7f2d617fa700 1 -- 172.21.15.123:0/4143521728 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d58041040 con 0x7f2d640f5410 2024-01-13T11:55:29.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.025+0000 7f2d63fff700 1 --2- 172.21.15.123:0/4143521728 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f2d5402ff60 0x7f2d54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:29.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.025+0000 7f2d63fff700 1 --2- 172.21.15.123:0/4143521728 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f2d5402ff60 0x7f2d54032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f2d500096a0 tx=0x7f2d50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:29.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.030+0000 7f2d617fa700 1 -- 172.21.15.123:0/4143521728 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d580095f0 con 0x7f2d640f5410 2024-01-13T11:55:29.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.236+0000 7f2d6aad2700 1 -- 172.21.15.123:0/4143521728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d6410a370 con 0x7f2d640f5410 2024-01-13T11:55:29.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.236+0000 7f2d617fa700 1 -- 172.21.15.123:0/4143521728 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d5801db70 con 0x7f2d640f5410 2024-01-13T11:55:29.224 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:29.225 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:29.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.238+0000 7f2d6aad2700 1 -- 172.21.15.123:0/4143521728 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f2d5402ff60 msgr2=0x7f2d54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:29.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.238+0000 7f2d6aad2700 1 --2- 172.21.15.123:0/4143521728 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f2d5402ff60 0x7f2d54032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f2d500096a0 tx=0x7f2d50006b40).stop 2024-01-13T11:55:29.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.238+0000 7f2d6aad2700 1 -- 172.21.15.123:0/4143521728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d640f5410 msgr2=0x7f2d641093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:29.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.238+0000 7f2d6aad2700 1 --2- 172.21.15.123:0/4143521728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d640f5410 0x7f2d641093f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f2d58013510 tx=0x7f2d5800b040).stop 2024-01-13T11:55:29.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.238+0000 7f2d6aad2700 1 -- 172.21.15.123:0/4143521728 shutdown_connections 2024-01-13T11:55:29.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.238+0000 7f2d6aad2700 1 --2- 172.21.15.123:0/4143521728 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f2d5402ff60 0x7f2d54032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:29.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.238+0000 7f2d6aad2700 1 --2- 172.21.15.123:0/4143521728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d640f5410 0x7f2d641093f0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:29.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.238+0000 7f2d6aad2700 1 -- 172.21.15.123:0/4143521728 >> 172.21.15.123:0/4143521728 conn(0x7f2d640f0b70 msgr2=0x7f2d640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:29.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.238+0000 7f2d6aad2700 1 -- 172.21.15.123:0/4143521728 shutdown_connections 2024-01-13T11:55:29.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:29.238+0000 7f2d6aad2700 1 -- 172.21.15.123:0/4143521728 wait complete. 2024-01-13T11:55:29.227 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:29.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:29 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:27.554874+0000 mgr.smithi059.hmyldj (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:30.613 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:30.614 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:30.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:30 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:29.237650+0000 mon.smithi059 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.123:0/4143521728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:31.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:31 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:29.555204+0000 mgr.smithi059.hmyldj (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:31.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.789+0000 7efd8c350700 1 -- 172.21.15.123:0/629780279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd840f6510 msgr2=0x7efd840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:31.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.789+0000 7efd8c350700 1 --2- 172.21.15.123:0/629780279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd840f6510 0x7efd840f6930 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7efd74009fb0 tx=0x7efd74009b30).stop 2024-01-13T11:55:31.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.789+0000 7efd8c350700 1 -- 172.21.15.123:0/629780279 shutdown_connections 2024-01-13T11:55:31.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.789+0000 7efd8c350700 1 --2- 172.21.15.123:0/629780279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd840f6510 0x7efd840f6930 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:31.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.789+0000 7efd8c350700 1 -- 172.21.15.123:0/629780279 >> 172.21.15.123:0/629780279 conn(0x7efd840f1c70 msgr2=0x7efd840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:31.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.789+0000 7efd8c350700 1 -- 172.21.15.123:0/629780279 shutdown_connections 2024-01-13T11:55:31.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.789+0000 7efd8c350700 1 -- 172.21.15.123:0/629780279 wait complete. 2024-01-13T11:55:31.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.790+0000 7efd8c350700 1 Processor -- start 2024-01-13T11:55:31.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.790+0000 7efd8c350700 1 -- start start 2024-01-13T11:55:31.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.790+0000 7efd8c350700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd840f6510 0x7efd84106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:31.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.790+0000 7efd8c350700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd84106c20 con 0x7efd840f6510 2024-01-13T11:55:31.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.790+0000 7efd8a0ec700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd840f6510 0x7efd84106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:31.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.790+0000 7efd8a0ec700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd840f6510 0x7efd84106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:39352/0 (socket says 172.21.15.123:39352) 2024-01-13T11:55:31.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.790+0000 7efd8a0ec700 1 -- 172.21.15.123:0/2918552590 learned_addr learned my addr 172.21.15.123:0/2918552590 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:31.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.791+0000 7efd8a0ec700 1 -- 172.21.15.123:0/2918552590 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd74005040 con 0x7efd840f6510 2024-01-13T11:55:31.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.791+0000 7efd8a0ec700 1 --2- 172.21.15.123:0/2918552590 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd840f6510 0x7efd84106730 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7efd74013510 tx=0x7efd7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:31.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.791+0000 7efd7affd700 1 -- 172.21.15.123:0/2918552590 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd74009320 con 0x7efd840f6510 2024-01-13T11:55:31.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.791+0000 7efd8c350700 1 -- 172.21.15.123:0/2918552590 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd84106e20 con 0x7efd840f6510 2024-01-13T11:55:31.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.791+0000 7efd7affd700 1 -- 172.21.15.123:0/2918552590 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd74013f00 con 0x7efd840f6510 2024-01-13T11:55:31.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.791+0000 7efd7affd700 1 -- 172.21.15.123:0/2918552590 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd74014d90 con 0x7efd840f6510 2024-01-13T11:55:31.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.791+0000 7efd8c350700 1 -- 172.21.15.123:0/2918552590 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd841072b0 con 0x7efd840f6510 2024-01-13T11:55:31.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.792+0000 7efd7affd700 1 -- 172.21.15.123:0/2918552590 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efd740095f0 con 0x7efd840f6510 2024-01-13T11:55:31.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.792+0000 7efd8c350700 1 -- 172.21.15.123:0/2918552590 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd84045ad0 con 0x7efd840f6510 2024-01-13T11:55:31.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.792+0000 7efd7affd700 1 --2- 172.21.15.123:0/2918552590 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7efd7002ff60 0x7efd70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:31.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.792+0000 7efd7affd700 1 -- 172.21.15.123:0/2918552590 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efd74040fc0 con 0x7efd840f6510 2024-01-13T11:55:31.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.792+0000 7efd898eb700 1 --2- 172.21.15.123:0/2918552590 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7efd7002ff60 0x7efd70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:31.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.792+0000 7efd898eb700 1 --2- 172.21.15.123:0/2918552590 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7efd7002ff60 0x7efd70032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7efd800096a0 tx=0x7efd80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:31.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:31.797+0000 7efd7affd700 1 -- 172.21.15.123:0/2918552590 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efd74014420 con 0x7efd840f6510 2024-01-13T11:55:31.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:32.002+0000 7efd8c350700 1 -- 172.21.15.123:0/2918552590 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efd840fbf60 con 0x7efd840f6510 2024-01-13T11:55:31.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:32.002+0000 7efd7affd700 1 -- 172.21.15.123:0/2918552590 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efd74010230 con 0x7efd840f6510 2024-01-13T11:55:31.990 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:31.990 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:31.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:32.004+0000 7efd8c350700 1 -- 172.21.15.123:0/2918552590 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7efd7002ff60 msgr2=0x7efd70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:31.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:32.004+0000 7efd8c350700 1 --2- 172.21.15.123:0/2918552590 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7efd7002ff60 0x7efd70032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7efd800096a0 tx=0x7efd80006b40).stop 2024-01-13T11:55:31.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:32.004+0000 7efd8c350700 1 -- 172.21.15.123:0/2918552590 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd840f6510 msgr2=0x7efd84106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:31.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:32.004+0000 7efd8c350700 1 --2- 172.21.15.123:0/2918552590 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd840f6510 0x7efd84106730 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7efd74013510 tx=0x7efd7400b040).stop 2024-01-13T11:55:31.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:32.004+0000 7efd8c350700 1 -- 172.21.15.123:0/2918552590 shutdown_connections 2024-01-13T11:55:31.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:32.004+0000 7efd8c350700 1 --2- 172.21.15.123:0/2918552590 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7efd7002ff60 0x7efd70032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:31.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:32.004+0000 7efd8c350700 1 --2- 172.21.15.123:0/2918552590 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd840f6510 0x7efd84106730 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:31.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:32.004+0000 7efd8c350700 1 -- 172.21.15.123:0/2918552590 >> 172.21.15.123:0/2918552590 conn(0x7efd840f1c70 msgr2=0x7efd840f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:31.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:32.004+0000 7efd8c350700 1 -- 172.21.15.123:0/2918552590 shutdown_connections 2024-01-13T11:55:31.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:32.004+0000 7efd8c350700 1 -- 172.21.15.123:0/2918552590 wait complete. 2024-01-13T11:55:31.993 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:32.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:32 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:32.003789+0000 mon.smithi059 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.123:0/2918552590' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:33.387 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:33.387 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:33.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:33 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:31.555593+0000 mgr.smithi059.hmyldj (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:34.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.597+0000 7f6d01da2700 1 -- 172.21.15.123:0/2753697897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cfc060280 msgr2=0x7f6cfc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:34.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.597+0000 7f6d01da2700 1 --2- 172.21.15.123:0/2753697897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cfc060280 0x7f6cfc0f90f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f6cf0009fb0 tx=0x7f6cf0009b30).stop 2024-01-13T11:55:34.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.597+0000 7f6d01da2700 1 -- 172.21.15.123:0/2753697897 shutdown_connections 2024-01-13T11:55:34.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.597+0000 7f6d01da2700 1 --2- 172.21.15.123:0/2753697897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cfc060280 0x7f6cfc0f90f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:34.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.597+0000 7f6d01da2700 1 -- 172.21.15.123:0/2753697897 >> 172.21.15.123:0/2753697897 conn(0x7f6cfc0f0b70 msgr2=0x7f6cfc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:34.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.597+0000 7f6d01da2700 1 -- 172.21.15.123:0/2753697897 shutdown_connections 2024-01-13T11:55:34.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.597+0000 7f6d01da2700 1 -- 172.21.15.123:0/2753697897 wait complete. 2024-01-13T11:55:34.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.598+0000 7f6d01da2700 1 Processor -- start 2024-01-13T11:55:34.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.598+0000 7f6d01da2700 1 -- start start 2024-01-13T11:55:34.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.598+0000 7f6d01da2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cfc060280 0x7f6cfc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:34.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.598+0000 7f6d01da2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc106bd0 con 0x7f6cfc060280 2024-01-13T11:55:34.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.599+0000 7f6cfb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cfc060280 0x7f6cfc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:34.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.599+0000 7f6cfb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cfc060280 0x7f6cfc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:39372/0 (socket says 172.21.15.123:39372) 2024-01-13T11:55:34.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.599+0000 7f6cfb7fe700 1 -- 172.21.15.123:0/765342976 learned_addr learned my addr 172.21.15.123:0/765342976 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:34.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.599+0000 7f6cfb7fe700 1 -- 172.21.15.123:0/765342976 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf0005040 con 0x7f6cfc060280 2024-01-13T11:55:34.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.599+0000 7f6cfb7fe700 1 --2- 172.21.15.123:0/765342976 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cfc060280 0x7f6cfc1066e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f6cf0013510 tx=0x7f6cf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:34.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.599+0000 7f6ce3fff700 1 -- 172.21.15.123:0/765342976 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cf0009320 con 0x7f6cfc060280 2024-01-13T11:55:34.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.599+0000 7f6d01da2700 1 -- 172.21.15.123:0/765342976 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cfc106dd0 con 0x7f6cfc060280 2024-01-13T11:55:34.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.599+0000 7f6ce3fff700 1 -- 172.21.15.123:0/765342976 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6cf0013f00 con 0x7f6cfc060280 2024-01-13T11:55:34.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.599+0000 7f6ce3fff700 1 -- 172.21.15.123:0/765342976 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cf0014dc0 con 0x7f6cfc060280 2024-01-13T11:55:34.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.599+0000 7f6d01da2700 1 -- 172.21.15.123:0/765342976 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cfc107260 con 0x7f6cfc060280 2024-01-13T11:55:34.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.600+0000 7f6ce3fff700 1 -- 172.21.15.123:0/765342976 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6cf00095f0 con 0x7f6cfc060280 2024-01-13T11:55:34.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.600+0000 7f6d01da2700 1 -- 172.21.15.123:0/765342976 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cfc045ad0 con 0x7f6cfc060280 2024-01-13T11:55:34.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.600+0000 7f6ce3fff700 1 --2- 172.21.15.123:0/765342976 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6ce402ff60 0x7f6ce4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:34.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.600+0000 7f6ce3fff700 1 -- 172.21.15.123:0/765342976 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6cf0041000 con 0x7f6cfc060280 2024-01-13T11:55:34.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.601+0000 7f6cfaffd700 1 --2- 172.21.15.123:0/765342976 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6ce402ff60 0x7f6ce4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:34.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.601+0000 7f6cfaffd700 1 --2- 172.21.15.123:0/765342976 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6ce402ff60 0x7f6ce4032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f6cec0096a0 tx=0x7f6cec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:34.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.605+0000 7f6ce3fff700 1 -- 172.21.15.123:0/765342976 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6cf001c790 con 0x7f6cfc060280 2024-01-13T11:55:34.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.811+0000 7f6d01da2700 1 -- 172.21.15.123:0/765342976 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6cfc108110 con 0x7f6cfc060280 2024-01-13T11:55:34.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.812+0000 7f6ce3fff700 1 -- 172.21.15.123:0/765342976 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6cf0010230 con 0x7f6cfc060280 2024-01-13T11:55:34.800 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:34.800 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:34.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.813+0000 7f6d01da2700 1 -- 172.21.15.123:0/765342976 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6ce402ff60 msgr2=0x7f6ce4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:34.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.813+0000 7f6d01da2700 1 --2- 172.21.15.123:0/765342976 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6ce402ff60 0x7f6ce4032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f6cec0096a0 tx=0x7f6cec006b40).stop 2024-01-13T11:55:34.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.813+0000 7f6d01da2700 1 -- 172.21.15.123:0/765342976 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cfc060280 msgr2=0x7f6cfc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:34.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.813+0000 7f6d01da2700 1 --2- 172.21.15.123:0/765342976 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cfc060280 0x7f6cfc1066e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f6cf0013510 tx=0x7f6cf000b040).stop 2024-01-13T11:55:34.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.813+0000 7f6d01da2700 1 -- 172.21.15.123:0/765342976 shutdown_connections 2024-01-13T11:55:34.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.813+0000 7f6d01da2700 1 --2- 172.21.15.123:0/765342976 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6ce402ff60 0x7f6ce4032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:34.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.813+0000 7f6d01da2700 1 --2- 172.21.15.123:0/765342976 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cfc060280 0x7f6cfc1066e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:34.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.813+0000 7f6d01da2700 1 -- 172.21.15.123:0/765342976 >> 172.21.15.123:0/765342976 conn(0x7f6cfc0f0b70 msgr2=0x7f6cfc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:34.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.814+0000 7f6d01da2700 1 -- 172.21.15.123:0/765342976 shutdown_connections 2024-01-13T11:55:34.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:34.814+0000 7f6d01da2700 1 -- 172.21.15.123:0/765342976 wait complete. 2024-01-13T11:55:34.802 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:35.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:35 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:33.555884+0000 mgr.smithi059.hmyldj (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:35.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:35 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:34.813171+0000 mon.smithi059 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.123:0/765342976' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:36.171 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:36.171 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:37.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.384+0000 7f8f1f13f700 1 -- 172.21.15.123:0/1196479965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f180f51a0 msgr2=0x7f8f180f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:37.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.384+0000 7f8f1f13f700 1 --2- 172.21.15.123:0/1196479965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f180f51a0 0x7f8f180f55c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f8f08009fb0 tx=0x7f8f08009b30).stop 2024-01-13T11:55:37.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.384+0000 7f8f1f13f700 1 -- 172.21.15.123:0/1196479965 shutdown_connections 2024-01-13T11:55:37.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.384+0000 7f8f1f13f700 1 --2- 172.21.15.123:0/1196479965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f180f51a0 0x7f8f180f55c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:37.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.384+0000 7f8f1f13f700 1 -- 172.21.15.123:0/1196479965 >> 172.21.15.123:0/1196479965 conn(0x7f8f180f09a0 msgr2=0x7f8f180f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:37.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.384+0000 7f8f1f13f700 1 -- 172.21.15.123:0/1196479965 shutdown_connections 2024-01-13T11:55:37.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.384+0000 7f8f1f13f700 1 -- 172.21.15.123:0/1196479965 wait complete. 2024-01-13T11:55:37.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.384+0000 7f8f1f13f700 1 Processor -- start 2024-01-13T11:55:37.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.384+0000 7f8f1f13f700 1 -- start start 2024-01-13T11:55:37.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.384+0000 7f8f1f13f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f180f51a0 0x7f8f181064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:37.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.384+0000 7f8f1f13f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f18106990 con 0x7f8f180f51a0 2024-01-13T11:55:37.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.385+0000 7f8f1e13d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f180f51a0 0x7f8f181064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:37.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.385+0000 7f8f1e13d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f180f51a0 0x7f8f181064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:46142/0 (socket says 172.21.15.123:46142) 2024-01-13T11:55:37.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.385+0000 7f8f1e13d700 1 -- 172.21.15.123:0/4123372933 learned_addr learned my addr 172.21.15.123:0/4123372933 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:37.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.385+0000 7f8f1e13d700 1 -- 172.21.15.123:0/4123372933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f08005040 con 0x7f8f180f51a0 2024-01-13T11:55:37.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.385+0000 7f8f1e13d700 1 --2- 172.21.15.123:0/4123372933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f180f51a0 0x7f8f181064a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f8f08013510 tx=0x7f8f0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:37.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.385+0000 7f8f0effd700 1 -- 172.21.15.123:0/4123372933 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f08009320 con 0x7f8f180f51a0 2024-01-13T11:55:37.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.385+0000 7f8f1f13f700 1 -- 172.21.15.123:0/4123372933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f18106b90 con 0x7f8f180f51a0 2024-01-13T11:55:37.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.385+0000 7f8f0effd700 1 -- 172.21.15.123:0/4123372933 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f08013f00 con 0x7f8f180f51a0 2024-01-13T11:55:37.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.385+0000 7f8f0effd700 1 -- 172.21.15.123:0/4123372933 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f08014dc0 con 0x7f8f180f51a0 2024-01-13T11:55:37.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.385+0000 7f8f1f13f700 1 -- 172.21.15.123:0/4123372933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f18107020 con 0x7f8f180f51a0 2024-01-13T11:55:37.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.386+0000 7f8f0effd700 1 -- 172.21.15.123:0/4123372933 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f080095f0 con 0x7f8f180f51a0 2024-01-13T11:55:37.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.386+0000 7f8f1f13f700 1 -- 172.21.15.123:0/4123372933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f18045ad0 con 0x7f8f180f51a0 2024-01-13T11:55:37.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.386+0000 7f8f0effd700 1 --2- 172.21.15.123:0/4123372933 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8f0402ff60 0x7f8f04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:37.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.386+0000 7f8f0effd700 1 -- 172.21.15.123:0/4123372933 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f08041000 con 0x7f8f180f51a0 2024-01-13T11:55:37.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.386+0000 7f8f1d93c700 1 --2- 172.21.15.123:0/4123372933 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8f0402ff60 0x7f8f04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:37.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.387+0000 7f8f1d93c700 1 --2- 172.21.15.123:0/4123372933 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8f0402ff60 0x7f8f04032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f8f140096a0 tx=0x7f8f14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:37.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.391+0000 7f8f0effd700 1 -- 172.21.15.123:0/4123372933 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f08014420 con 0x7f8f180f51a0 2024-01-13T11:55:37.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.599+0000 7f8f1f13f700 1 -- 172.21.15.123:0/4123372933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f18107ed0 con 0x7f8f180f51a0 2024-01-13T11:55:37.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.600+0000 7f8f0effd700 1 -- 172.21.15.123:0/4123372933 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8f08010230 con 0x7f8f180f51a0 2024-01-13T11:55:37.587 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:37.588 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:37.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.601+0000 7f8f1f13f700 1 -- 172.21.15.123:0/4123372933 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8f0402ff60 msgr2=0x7f8f04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:37.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.601+0000 7f8f1f13f700 1 --2- 172.21.15.123:0/4123372933 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8f0402ff60 0x7f8f04032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f8f140096a0 tx=0x7f8f14006b40).stop 2024-01-13T11:55:37.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.601+0000 7f8f1f13f700 1 -- 172.21.15.123:0/4123372933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f180f51a0 msgr2=0x7f8f181064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:37.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.601+0000 7f8f1f13f700 1 --2- 172.21.15.123:0/4123372933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f180f51a0 0x7f8f181064a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f8f08013510 tx=0x7f8f0800b040).stop 2024-01-13T11:55:37.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.601+0000 7f8f1f13f700 1 -- 172.21.15.123:0/4123372933 shutdown_connections 2024-01-13T11:55:37.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.601+0000 7f8f1f13f700 1 --2- 172.21.15.123:0/4123372933 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8f0402ff60 0x7f8f04032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:37.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.601+0000 7f8f1f13f700 1 --2- 172.21.15.123:0/4123372933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f180f51a0 0x7f8f181064a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:37.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.601+0000 7f8f1f13f700 1 -- 172.21.15.123:0/4123372933 >> 172.21.15.123:0/4123372933 conn(0x7f8f180f09a0 msgr2=0x7f8f1805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:37.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.601+0000 7f8f1f13f700 1 -- 172.21.15.123:0/4123372933 shutdown_connections 2024-01-13T11:55:37.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:37.601+0000 7f8f1f13f700 1 -- 172.21.15.123:0/4123372933 wait complete. 2024-01-13T11:55:37.590 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:37.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:37 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:35.556197+0000 mgr.smithi059.hmyldj (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:38.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:38 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:37.600802+0000 mon.smithi059 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.123:0/4123372933' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:38.975 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:38.975 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:39.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:39 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:37.556487+0000 mgr.smithi059.hmyldj (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:40.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.157+0000 7f457155a700 1 -- 172.21.15.123:0/1928470002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f456c0f6510 msgr2=0x7f456c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.157+0000 7f457155a700 1 --2- 172.21.15.123:0/1928470002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f456c0f6510 0x7f456c0f6930 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f455c009fb0 tx=0x7f455c009b30).stop 2024-01-13T11:55:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.157+0000 7f457155a700 1 -- 172.21.15.123:0/1928470002 shutdown_connections 2024-01-13T11:55:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.157+0000 7f457155a700 1 --2- 172.21.15.123:0/1928470002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f456c0f6510 0x7f456c0f6930 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.157+0000 7f457155a700 1 -- 172.21.15.123:0/1928470002 >> 172.21.15.123:0/1928470002 conn(0x7f456c0f1c70 msgr2=0x7f456c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.157+0000 7f457155a700 1 -- 172.21.15.123:0/1928470002 shutdown_connections 2024-01-13T11:55:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.157+0000 7f457155a700 1 -- 172.21.15.123:0/1928470002 wait complete. 2024-01-13T11:55:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.158+0000 7f457155a700 1 Processor -- start 2024-01-13T11:55:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.158+0000 7f457155a700 1 -- start start 2024-01-13T11:55:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.158+0000 7f457155a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f456c0f6510 0x7f456c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.158+0000 7f457155a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456c069e20 con 0x7f456c0f6510 2024-01-13T11:55:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.158+0000 7f456affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f456c0f6510 0x7f456c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.158+0000 7f456affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f456c0f6510 0x7f456c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:46158/0 (socket says 172.21.15.123:46158) 2024-01-13T11:55:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.158+0000 7f456affd700 1 -- 172.21.15.123:0/2016571186 learned_addr learned my addr 172.21.15.123:0/2016571186 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.158+0000 7f456affd700 1 -- 172.21.15.123:0/2016571186 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455c005040 con 0x7f456c0f6510 2024-01-13T11:55:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.158+0000 7f456affd700 1 --2- 172.21.15.123:0/2016571186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f456c0f6510 0x7f456c06b9a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f455c013510 tx=0x7f455c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.159+0000 7f45537fe700 1 -- 172.21.15.123:0/2016571186 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f455c009320 con 0x7f456c0f6510 2024-01-13T11:55:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.159+0000 7f45537fe700 1 -- 172.21.15.123:0/2016571186 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f455c013f00 con 0x7f456c0f6510 2024-01-13T11:55:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.159+0000 7f457155a700 1 -- 172.21.15.123:0/2016571186 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f456c069fc0 con 0x7f456c0f6510 2024-01-13T11:55:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.159+0000 7f45537fe700 1 -- 172.21.15.123:0/2016571186 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f455c014d90 con 0x7f456c0f6510 2024-01-13T11:55:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.159+0000 7f457155a700 1 -- 172.21.15.123:0/2016571186 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f456c06a450 con 0x7f456c0f6510 2024-01-13T11:55:40.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.159+0000 7f45537fe700 1 -- 172.21.15.123:0/2016571186 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f455c014420 con 0x7f456c0f6510 2024-01-13T11:55:40.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.159+0000 7f457155a700 1 -- 172.21.15.123:0/2016571186 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f456c059100 con 0x7f456c0f6510 2024-01-13T11:55:40.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.160+0000 7f45537fe700 1 --2- 172.21.15.123:0/2016571186 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f455402ff60 0x7f4554032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:40.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.160+0000 7f45537fe700 1 -- 172.21.15.123:0/2016571186 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f455c041070 con 0x7f456c0f6510 2024-01-13T11:55:40.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.160+0000 7f456a7fc700 1 --2- 172.21.15.123:0/2016571186 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f455402ff60 0x7f4554032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:40.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.160+0000 7f456a7fc700 1 --2- 172.21.15.123:0/2016571186 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f455402ff60 0x7f4554032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f45600096a0 tx=0x7f4560006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:40.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.164+0000 7f45537fe700 1 -- 172.21.15.123:0/2016571186 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f455c0095f0 con 0x7f456c0f6510 2024-01-13T11:55:40.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.372+0000 7f457155a700 1 -- 172.21.15.123:0/2016571186 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f456c0fbf60 con 0x7f456c0f6510 2024-01-13T11:55:40.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.373+0000 7f45537fe700 1 -- 172.21.15.123:0/2016571186 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f455c01db70 con 0x7f456c0f6510 2024-01-13T11:55:40.361 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:40.361 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:40.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.375+0000 7f457155a700 1 -- 172.21.15.123:0/2016571186 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f455402ff60 msgr2=0x7f4554032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:40.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.375+0000 7f457155a700 1 --2- 172.21.15.123:0/2016571186 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f455402ff60 0x7f4554032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f45600096a0 tx=0x7f4560006b40).stop 2024-01-13T11:55:40.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.375+0000 7f457155a700 1 -- 172.21.15.123:0/2016571186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f456c0f6510 msgr2=0x7f456c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:40.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.375+0000 7f457155a700 1 --2- 172.21.15.123:0/2016571186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f456c0f6510 0x7f456c06b9a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f455c013510 tx=0x7f455c00b040).stop 2024-01-13T11:55:40.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.375+0000 7f457155a700 1 -- 172.21.15.123:0/2016571186 shutdown_connections 2024-01-13T11:55:40.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.375+0000 7f457155a700 1 --2- 172.21.15.123:0/2016571186 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f455402ff60 0x7f4554032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:40.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.375+0000 7f457155a700 1 --2- 172.21.15.123:0/2016571186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f456c0f6510 0x7f456c06b9a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:40.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.375+0000 7f457155a700 1 -- 172.21.15.123:0/2016571186 >> 172.21.15.123:0/2016571186 conn(0x7f456c0f1c70 msgr2=0x7f456c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:40.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.375+0000 7f457155a700 1 -- 172.21.15.123:0/2016571186 shutdown_connections 2024-01-13T11:55:40.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:40.375+0000 7f457155a700 1 -- 172.21.15.123:0/2016571186 wait complete. 2024-01-13T11:55:40.364 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:40.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:39.643101+0000 mon.smithi059 (mon.0) 253 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/mirror_snapshot_schedule"}]: dispatch 2024-01-13T11:55:40.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:39.643930+0000 mon.smithi059 (mon.0) 254 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/trash_purge_schedule"}]: dispatch 2024-01-13T11:55:41.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:41 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:39.556790+0000 mgr.smithi059.hmyldj (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:41.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:41 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:40.374319+0000 mon.smithi059 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.123:0/2016571186' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:41.767 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:41.768 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:42.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.954+0000 7f6c29e4e700 1 -- 172.21.15.123:0/476531536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c240f5410 msgr2=0x7f6c240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:42.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.954+0000 7f6c29e4e700 1 --2- 172.21.15.123:0/476531536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c240f5410 0x7f6c240f5830 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f6c14009fb0 tx=0x7f6c14009b30).stop 2024-01-13T11:55:42.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.954+0000 7f6c29e4e700 1 -- 172.21.15.123:0/476531536 shutdown_connections 2024-01-13T11:55:42.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.954+0000 7f6c29e4e700 1 --2- 172.21.15.123:0/476531536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c240f5410 0x7f6c240f5830 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:42.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.954+0000 7f6c29e4e700 1 -- 172.21.15.123:0/476531536 >> 172.21.15.123:0/476531536 conn(0x7f6c240f0b70 msgr2=0x7f6c240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:42.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.954+0000 7f6c29e4e700 1 -- 172.21.15.123:0/476531536 shutdown_connections 2024-01-13T11:55:42.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.954+0000 7f6c29e4e700 1 -- 172.21.15.123:0/476531536 wait complete. 2024-01-13T11:55:42.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.955+0000 7f6c29e4e700 1 Processor -- start 2024-01-13T11:55:42.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.955+0000 7f6c29e4e700 1 -- start start 2024-01-13T11:55:42.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.955+0000 7f6c29e4e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c240f5410 0x7f6c24109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:42.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.955+0000 7f6c29e4e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c24060240 con 0x7f6c240f5410 2024-01-13T11:55:42.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.955+0000 7f6c237fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c240f5410 0x7f6c24109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:42.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.955+0000 7f6c237fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c240f5410 0x7f6c24109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:46174/0 (socket says 172.21.15.123:46174) 2024-01-13T11:55:42.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.955+0000 7f6c237fe700 1 -- 172.21.15.123:0/2184737058 learned_addr learned my addr 172.21.15.123:0/2184737058 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:42.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.956+0000 7f6c237fe700 1 -- 172.21.15.123:0/2184737058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c14005040 con 0x7f6c240f5410 2024-01-13T11:55:42.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.956+0000 7f6c237fe700 1 --2- 172.21.15.123:0/2184737058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c240f5410 0x7f6c24109430 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f6c14013510 tx=0x7f6c1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:42.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.956+0000 7f6c0bfff700 1 -- 172.21.15.123:0/2184737058 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c14009320 con 0x7f6c240f5410 2024-01-13T11:55:42.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.956+0000 7f6c29e4e700 1 -- 172.21.15.123:0/2184737058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c24060440 con 0x7f6c240f5410 2024-01-13T11:55:42.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.956+0000 7f6c0bfff700 1 -- 172.21.15.123:0/2184737058 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c14013f00 con 0x7f6c240f5410 2024-01-13T11:55:42.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.956+0000 7f6c0bfff700 1 -- 172.21.15.123:0/2184737058 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c14014d90 con 0x7f6c240f5410 2024-01-13T11:55:42.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.956+0000 7f6c29e4e700 1 -- 172.21.15.123:0/2184737058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c24109ae0 con 0x7f6c240f5410 2024-01-13T11:55:42.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.957+0000 7f6c0bfff700 1 -- 172.21.15.123:0/2184737058 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c140095f0 con 0x7f6c240f5410 2024-01-13T11:55:42.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.957+0000 7f6c29e4e700 1 -- 172.21.15.123:0/2184737058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c24045ad0 con 0x7f6c240f5410 2024-01-13T11:55:42.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.957+0000 7f6c0bfff700 1 --2- 172.21.15.123:0/2184737058 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6c0c02ff60 0x7f6c0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:42.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.957+0000 7f6c0bfff700 1 -- 172.21.15.123:0/2184737058 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c14040fe0 con 0x7f6c240f5410 2024-01-13T11:55:42.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.957+0000 7f6c22ffd700 1 --2- 172.21.15.123:0/2184737058 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6c0c02ff60 0x7f6c0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:42.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.958+0000 7f6c22ffd700 1 --2- 172.21.15.123:0/2184737058 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6c0c02ff60 0x7f6c0c032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f6c180096a0 tx=0x7f6c18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:42.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:42.962+0000 7f6c0bfff700 1 -- 172.21.15.123:0/2184737058 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c1401d7f0 con 0x7f6c240f5410 2024-01-13T11:55:43.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:43.168+0000 7f6c29e4e700 1 -- 172.21.15.123:0/2184737058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c240fae60 con 0x7f6c240f5410 2024-01-13T11:55:43.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:43.169+0000 7f6c0bfff700 1 -- 172.21.15.123:0/2184737058 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6c14010230 con 0x7f6c240f5410 2024-01-13T11:55:43.157 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:43.157 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:43.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:43.171+0000 7f6c29e4e700 1 -- 172.21.15.123:0/2184737058 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6c0c02ff60 msgr2=0x7f6c0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:43.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:43.171+0000 7f6c29e4e700 1 --2- 172.21.15.123:0/2184737058 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6c0c02ff60 0x7f6c0c032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f6c180096a0 tx=0x7f6c18006b40).stop 2024-01-13T11:55:43.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:43.171+0000 7f6c29e4e700 1 -- 172.21.15.123:0/2184737058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c240f5410 msgr2=0x7f6c24109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:43.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:43.171+0000 7f6c29e4e700 1 --2- 172.21.15.123:0/2184737058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c240f5410 0x7f6c24109430 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f6c14013510 tx=0x7f6c1400b040).stop 2024-01-13T11:55:43.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:43.171+0000 7f6c29e4e700 1 -- 172.21.15.123:0/2184737058 shutdown_connections 2024-01-13T11:55:43.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:43.171+0000 7f6c29e4e700 1 --2- 172.21.15.123:0/2184737058 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6c0c02ff60 0x7f6c0c032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:43.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:43.171+0000 7f6c29e4e700 1 --2- 172.21.15.123:0/2184737058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c240f5410 0x7f6c24109430 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:43.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:43.171+0000 7f6c29e4e700 1 -- 172.21.15.123:0/2184737058 >> 172.21.15.123:0/2184737058 conn(0x7f6c240f0b70 msgr2=0x7f6c240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:43.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:43.171+0000 7f6c29e4e700 1 -- 172.21.15.123:0/2184737058 shutdown_connections 2024-01-13T11:55:43.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:43.171+0000 7f6c29e4e700 1 -- 172.21.15.123:0/2184737058 wait complete. 2024-01-13T11:55:43.160 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:43.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:43 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:41.557197+0000 mgr.smithi059.hmyldj (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:43.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:43 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:43.170240+0000 mon.smithi059 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.123:0/2184737058' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:44.554 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:44.554 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:45.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:45 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:43.557593+0000 mgr.smithi059.hmyldj (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:45.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.690+0000 7fca84a7e700 1 -- 172.21.15.123:0/3061103983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca800f5410 msgr2=0x7fca800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:45.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.690+0000 7fca84a7e700 1 --2- 172.21.15.123:0/3061103983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca800f5410 0x7fca800f5830 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fca74009fb0 tx=0x7fca74009b30).stop 2024-01-13T11:55:45.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.690+0000 7fca84a7e700 1 -- 172.21.15.123:0/3061103983 shutdown_connections 2024-01-13T11:55:45.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.690+0000 7fca84a7e700 1 --2- 172.21.15.123:0/3061103983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca800f5410 0x7fca800f5830 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:45.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.690+0000 7fca84a7e700 1 -- 172.21.15.123:0/3061103983 >> 172.21.15.123:0/3061103983 conn(0x7fca800f0b70 msgr2=0x7fca800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:45.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.690+0000 7fca84a7e700 1 -- 172.21.15.123:0/3061103983 shutdown_connections 2024-01-13T11:55:45.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.690+0000 7fca84a7e700 1 -- 172.21.15.123:0/3061103983 wait complete. 2024-01-13T11:55:45.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.691+0000 7fca84a7e700 1 Processor -- start 2024-01-13T11:55:45.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.691+0000 7fca84a7e700 1 -- start start 2024-01-13T11:55:45.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.691+0000 7fca84a7e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca800f5410 0x7fca80106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:45.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.691+0000 7fca84a7e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca80106c20 con 0x7fca800f5410 2024-01-13T11:55:45.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.691+0000 7fca7e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca800f5410 0x7fca80106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:45.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.691+0000 7fca7e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca800f5410 0x7fca80106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:46202/0 (socket says 172.21.15.123:46202) 2024-01-13T11:55:45.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.691+0000 7fca7e59c700 1 -- 172.21.15.123:0/697488151 learned_addr learned my addr 172.21.15.123:0/697488151 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:45.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.691+0000 7fca7e59c700 1 -- 172.21.15.123:0/697488151 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca74005040 con 0x7fca800f5410 2024-01-13T11:55:45.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.691+0000 7fca7e59c700 1 --2- 172.21.15.123:0/697488151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca800f5410 0x7fca80106730 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fca74013510 tx=0x7fca7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:45.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.692+0000 7fca6effd700 1 -- 172.21.15.123:0/697488151 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca74009320 con 0x7fca800f5410 2024-01-13T11:55:45.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.692+0000 7fca84a7e700 1 -- 172.21.15.123:0/697488151 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca80106e20 con 0x7fca800f5410 2024-01-13T11:55:45.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.692+0000 7fca6effd700 1 -- 172.21.15.123:0/697488151 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca74013f00 con 0x7fca800f5410 2024-01-13T11:55:45.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.692+0000 7fca6effd700 1 -- 172.21.15.123:0/697488151 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca74014d90 con 0x7fca800f5410 2024-01-13T11:55:45.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.692+0000 7fca84a7e700 1 -- 172.21.15.123:0/697488151 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca801072b0 con 0x7fca800f5410 2024-01-13T11:55:45.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.692+0000 7fca6effd700 1 -- 172.21.15.123:0/697488151 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca740095f0 con 0x7fca800f5410 2024-01-13T11:55:45.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.692+0000 7fca84a7e700 1 -- 172.21.15.123:0/697488151 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca80045ad0 con 0x7fca800f5410 2024-01-13T11:55:45.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.693+0000 7fca6effd700 1 --2- 172.21.15.123:0/697488151 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fca6802ff60 0x7fca68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:45.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.693+0000 7fca6effd700 1 -- 172.21.15.123:0/697488151 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca74041000 con 0x7fca800f5410 2024-01-13T11:55:45.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.693+0000 7fca7dd9b700 1 --2- 172.21.15.123:0/697488151 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fca6802ff60 0x7fca68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:45.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.693+0000 7fca7dd9b700 1 --2- 172.21.15.123:0/697488151 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fca6802ff60 0x7fca68032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fca700096a0 tx=0x7fca70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:45.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.697+0000 7fca6effd700 1 -- 172.21.15.123:0/697488151 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca74014420 con 0x7fca800f5410 2024-01-13T11:55:45.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.904+0000 7fca84a7e700 1 -- 172.21.15.123:0/697488151 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca800fae60 con 0x7fca800f5410 2024-01-13T11:55:45.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.905+0000 7fca6effd700 1 -- 172.21.15.123:0/697488151 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fca7401d7f0 con 0x7fca800f5410 2024-01-13T11:55:45.893 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:45.893 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:45.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.906+0000 7fca84a7e700 1 -- 172.21.15.123:0/697488151 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fca6802ff60 msgr2=0x7fca68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:45.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.906+0000 7fca84a7e700 1 --2- 172.21.15.123:0/697488151 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fca6802ff60 0x7fca68032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fca700096a0 tx=0x7fca70006b40).stop 2024-01-13T11:55:45.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.906+0000 7fca84a7e700 1 -- 172.21.15.123:0/697488151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca800f5410 msgr2=0x7fca80106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:45.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.906+0000 7fca84a7e700 1 --2- 172.21.15.123:0/697488151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca800f5410 0x7fca80106730 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fca74013510 tx=0x7fca7400b040).stop 2024-01-13T11:55:45.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.906+0000 7fca84a7e700 1 -- 172.21.15.123:0/697488151 shutdown_connections 2024-01-13T11:55:45.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.906+0000 7fca84a7e700 1 --2- 172.21.15.123:0/697488151 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fca6802ff60 0x7fca68032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:45.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.906+0000 7fca84a7e700 1 --2- 172.21.15.123:0/697488151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca800f5410 0x7fca80106730 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:45.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.906+0000 7fca84a7e700 1 -- 172.21.15.123:0/697488151 >> 172.21.15.123:0/697488151 conn(0x7fca800f0b70 msgr2=0x7fca800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:45.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.906+0000 7fca84a7e700 1 -- 172.21.15.123:0/697488151 shutdown_connections 2024-01-13T11:55:45.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:45.906+0000 7fca84a7e700 1 -- 172.21.15.123:0/697488151 wait complete. 2024-01-13T11:55:45.895 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:46.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:46 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:45.905857+0000 mon.smithi059 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.123:0/697488151' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:47.287 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:47.287 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:47.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:47 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:45.557954+0000 mgr.smithi059.hmyldj (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:48.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.486+0000 7fc542689700 1 -- 172.21.15.123:0/122977086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 msgr2=0x7fc53c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:48.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.486+0000 7fc542689700 1 --2- 172.21.15.123:0/122977086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 0x7fc53c0f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fc52c009fb0 tx=0x7fc52c009b30).stop 2024-01-13T11:55:48.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.486+0000 7fc542689700 1 -- 172.21.15.123:0/122977086 shutdown_connections 2024-01-13T11:55:48.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.486+0000 7fc542689700 1 --2- 172.21.15.123:0/122977086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 0x7fc53c0f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:48.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.486+0000 7fc542689700 1 -- 172.21.15.123:0/122977086 >> 172.21.15.123:0/122977086 conn(0x7fc53c0f0b70 msgr2=0x7fc53c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:48.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.486+0000 7fc542689700 1 -- 172.21.15.123:0/122977086 shutdown_connections 2024-01-13T11:55:48.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.486+0000 7fc542689700 1 -- 172.21.15.123:0/122977086 wait complete. 2024-01-13T11:55:48.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.487+0000 7fc542689700 1 Processor -- start 2024-01-13T11:55:48.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.487+0000 7fc542689700 1 -- start start 2024-01-13T11:55:48.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.487+0000 7fc542689700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 0x7fc53c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:48.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.487+0000 7fc542689700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c060240 con 0x7fc53c0f5410 2024-01-13T11:55:48.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.487+0000 7fc53bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 0x7fc53c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:48.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.488+0000 7fc53bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 0x7fc53c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:53616/0 (socket says 172.21.15.123:53616) 2024-01-13T11:55:48.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.488+0000 7fc53bfff700 1 -- 172.21.15.123:0/859850431 learned_addr learned my addr 172.21.15.123:0/859850431 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:48.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.488+0000 7fc53bfff700 1 -- 172.21.15.123:0/859850431 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c005040 con 0x7fc53c0f5410 2024-01-13T11:55:48.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.488+0000 7fc53bfff700 1 --2- 172.21.15.123:0/859850431 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 0x7fc53c109430 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fc52c013510 tx=0x7fc52c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:48.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.488+0000 7fc538ff9700 1 -- 172.21.15.123:0/859850431 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc52c009320 con 0x7fc53c0f5410 2024-01-13T11:55:48.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.488+0000 7fc542689700 1 -- 172.21.15.123:0/859850431 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc53c060440 con 0x7fc53c0f5410 2024-01-13T11:55:48.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.488+0000 7fc538ff9700 1 -- 172.21.15.123:0/859850431 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc52c013f00 con 0x7fc53c0f5410 2024-01-13T11:55:48.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.488+0000 7fc538ff9700 1 -- 172.21.15.123:0/859850431 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc52c014d90 con 0x7fc53c0f5410 2024-01-13T11:55:48.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.488+0000 7fc542689700 1 -- 172.21.15.123:0/859850431 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc53c109ae0 con 0x7fc53c0f5410 2024-01-13T11:55:48.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.489+0000 7fc538ff9700 1 -- 172.21.15.123:0/859850431 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc52c0095f0 con 0x7fc53c0f5410 2024-01-13T11:55:48.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.489+0000 7fc542689700 1 -- 172.21.15.123:0/859850431 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc53c045ad0 con 0x7fc53c0f5410 2024-01-13T11:55:48.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.489+0000 7fc538ff9700 1 --2- 172.21.15.123:0/859850431 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc52802ff60 0x7fc528032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:48.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.489+0000 7fc538ff9700 1 -- 172.21.15.123:0/859850431 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc52c040fe0 con 0x7fc53c0f5410 2024-01-13T11:55:48.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.489+0000 7fc53b7fe700 1 --2- 172.21.15.123:0/859850431 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc52802ff60 0x7fc528032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:48.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.490+0000 7fc53b7fe700 1 --2- 172.21.15.123:0/859850431 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc52802ff60 0x7fc528032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fc5240096a0 tx=0x7fc524006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:48.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.494+0000 7fc538ff9700 1 -- 172.21.15.123:0/859850431 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc52c014420 con 0x7fc53c0f5410 2024-01-13T11:55:48.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.699+0000 7fc542689700 1 -- 172.21.15.123:0/859850431 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc53c0fae60 con 0x7fc53c0f5410 2024-01-13T11:55:48.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.700+0000 7fc538ff9700 1 -- 172.21.15.123:0/859850431 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc52c010230 con 0x7fc53c0f5410 2024-01-13T11:55:48.688 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:48.688 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:48.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.701+0000 7fc542689700 1 -- 172.21.15.123:0/859850431 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc52802ff60 msgr2=0x7fc528032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:48.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.701+0000 7fc542689700 1 --2- 172.21.15.123:0/859850431 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc52802ff60 0x7fc528032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fc5240096a0 tx=0x7fc524006b40).stop 2024-01-13T11:55:48.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.701+0000 7fc542689700 1 -- 172.21.15.123:0/859850431 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 msgr2=0x7fc53c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:48.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.701+0000 7fc542689700 1 --2- 172.21.15.123:0/859850431 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 0x7fc53c109430 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fc52c013510 tx=0x7fc52c00b040).stop 2024-01-13T11:55:48.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.701+0000 7fc542689700 1 -- 172.21.15.123:0/859850431 shutdown_connections 2024-01-13T11:55:48.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.701+0000 7fc542689700 1 --2- 172.21.15.123:0/859850431 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc52802ff60 0x7fc528032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:48.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.701+0000 7fc542689700 1 --2- 172.21.15.123:0/859850431 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 0x7fc53c109430 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:48.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.701+0000 7fc542689700 1 -- 172.21.15.123:0/859850431 >> 172.21.15.123:0/859850431 conn(0x7fc53c0f0b70 msgr2=0x7fc53c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:48.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.701+0000 7fc542689700 1 -- 172.21.15.123:0/859850431 shutdown_connections 2024-01-13T11:55:48.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:48.701+0000 7fc542689700 1 -- 172.21.15.123:0/859850431 wait complete. 2024-01-13T11:55:48.690 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:49.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:49 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:47.558281+0000 mgr.smithi059.hmyldj (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:49.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:49 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:48.700960+0000 mon.smithi059 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.123:0/859850431' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:50.073 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:50.074 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:51.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.238+0000 7f9bbc9fa700 1 -- 172.21.15.123:0/2419463332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bb80f53f0 msgr2=0x7f9bb80f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:51.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.238+0000 7f9bbc9fa700 1 --2- 172.21.15.123:0/2419463332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bb80f53f0 0x7f9bb80f5810 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f9ba0009fb0 tx=0x7f9ba0009b30).stop 2024-01-13T11:55:51.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.238+0000 7f9bbc9fa700 1 -- 172.21.15.123:0/2419463332 shutdown_connections 2024-01-13T11:55:51.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.238+0000 7f9bbc9fa700 1 --2- 172.21.15.123:0/2419463332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bb80f53f0 0x7f9bb80f5810 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:51.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.238+0000 7f9bbc9fa700 1 -- 172.21.15.123:0/2419463332 >> 172.21.15.123:0/2419463332 conn(0x7f9bb80f0b70 msgr2=0x7f9bb80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:51.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.238+0000 7f9bbc9fa700 1 -- 172.21.15.123:0/2419463332 shutdown_connections 2024-01-13T11:55:51.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.238+0000 7f9bbc9fa700 1 -- 172.21.15.123:0/2419463332 wait complete. 2024-01-13T11:55:51.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.238+0000 7f9bbc9fa700 1 Processor -- start 2024-01-13T11:55:51.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.238+0000 7f9bbc9fa700 1 -- start start 2024-01-13T11:55:51.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.238+0000 7f9bbc9fa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bb80f53f0 0x7f9bb8109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:51.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.238+0000 7f9bbc9fa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb8060220 con 0x7f9bb80f53f0 2024-01-13T11:55:51.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.239+0000 7f9bb659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bb80f53f0 0x7f9bb8109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:51.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.239+0000 7f9bb659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bb80f53f0 0x7f9bb8109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:53640/0 (socket says 172.21.15.123:53640) 2024-01-13T11:55:51.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.239+0000 7f9bb659c700 1 -- 172.21.15.123:0/2059169294 learned_addr learned my addr 172.21.15.123:0/2059169294 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:51.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.239+0000 7f9bb659c700 1 -- 172.21.15.123:0/2059169294 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba0005040 con 0x7f9bb80f53f0 2024-01-13T11:55:51.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.239+0000 7f9bb659c700 1 --2- 172.21.15.123:0/2059169294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bb80f53f0 0x7f9bb8109400 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f9ba0014000 tx=0x7f9ba000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:51.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.239+0000 7f9baeffd700 1 -- 172.21.15.123:0/2059169294 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ba00095f0 con 0x7f9bb80f53f0 2024-01-13T11:55:51.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.239+0000 7f9bbc9fa700 1 -- 172.21.15.123:0/2059169294 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb8060420 con 0x7f9bb80f53f0 2024-01-13T11:55:51.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.239+0000 7f9baeffd700 1 -- 172.21.15.123:0/2059169294 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ba0007e70 con 0x7f9bb80f53f0 2024-01-13T11:55:51.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.239+0000 7f9baeffd700 1 -- 172.21.15.123:0/2059169294 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ba0014ea0 con 0x7f9bb80f53f0 2024-01-13T11:55:51.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.239+0000 7f9bbc9fa700 1 -- 172.21.15.123:0/2059169294 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb8109ab0 con 0x7f9bb80f53f0 2024-01-13T11:55:51.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.240+0000 7f9baeffd700 1 -- 172.21.15.123:0/2059169294 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ba0014690 con 0x7f9bb80f53f0 2024-01-13T11:55:51.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.240+0000 7f9bbc9fa700 1 -- 172.21.15.123:0/2059169294 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb8059100 con 0x7f9bb80f53f0 2024-01-13T11:55:51.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.240+0000 7f9baeffd700 1 --2- 172.21.15.123:0/2059169294 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9ba402ff60 0x7f9ba4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:51.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.240+0000 7f9baeffd700 1 -- 172.21.15.123:0/2059169294 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ba0040870 con 0x7f9bb80f53f0 2024-01-13T11:55:51.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.241+0000 7f9bb5d9b700 1 --2- 172.21.15.123:0/2059169294 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9ba402ff60 0x7f9ba4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:51.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.241+0000 7f9bb5d9b700 1 --2- 172.21.15.123:0/2059169294 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9ba402ff60 0x7f9ba4032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f9ba80096a0 tx=0x7f9ba8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:51.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.245+0000 7f9baeffd700 1 -- 172.21.15.123:0/2059169294 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ba000f300 con 0x7f9bb80f53f0 2024-01-13T11:55:51.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.451+0000 7f9bbc9fa700 1 -- 172.21.15.123:0/2059169294 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9bb8045ad0 con 0x7f9bb80f53f0 2024-01-13T11:55:51.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.452+0000 7f9baeffd700 1 -- 172.21.15.123:0/2059169294 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ba000f300 con 0x7f9bb80f53f0 2024-01-13T11:55:51.440 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:51.440 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:51.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.453+0000 7f9bbc9fa700 1 -- 172.21.15.123:0/2059169294 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9ba402ff60 msgr2=0x7f9ba4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:51.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.453+0000 7f9bbc9fa700 1 --2- 172.21.15.123:0/2059169294 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9ba402ff60 0x7f9ba4032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f9ba80096a0 tx=0x7f9ba8006b40).stop 2024-01-13T11:55:51.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.453+0000 7f9bbc9fa700 1 -- 172.21.15.123:0/2059169294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bb80f53f0 msgr2=0x7f9bb8109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:51.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.453+0000 7f9bbc9fa700 1 --2- 172.21.15.123:0/2059169294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bb80f53f0 0x7f9bb8109400 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f9ba0014000 tx=0x7f9ba000b040).stop 2024-01-13T11:55:51.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.453+0000 7f9bbc9fa700 1 -- 172.21.15.123:0/2059169294 shutdown_connections 2024-01-13T11:55:51.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.453+0000 7f9bbc9fa700 1 --2- 172.21.15.123:0/2059169294 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9ba402ff60 0x7f9ba4032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:51.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.453+0000 7f9bbc9fa700 1 --2- 172.21.15.123:0/2059169294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bb80f53f0 0x7f9bb8109400 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:51.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.453+0000 7f9bbc9fa700 1 -- 172.21.15.123:0/2059169294 >> 172.21.15.123:0/2059169294 conn(0x7f9bb80f0b70 msgr2=0x7f9bb80f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:51.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.453+0000 7f9bbc9fa700 1 -- 172.21.15.123:0/2059169294 shutdown_connections 2024-01-13T11:55:51.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:51.453+0000 7f9bbc9fa700 1 -- 172.21.15.123:0/2059169294 wait complete. 2024-01-13T11:55:51.442 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:51.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:49.558618+0000 mgr.smithi059.hmyldj (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:52.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:51.452917+0000 mon.smithi059 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.123:0/2059169294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:52.814 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:52.815 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:53.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:53 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:51.558988+0000 mgr.smithi059.hmyldj (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:54.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.028+0000 7f4509bf8700 1 -- 172.21.15.123:0/2519562486 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45040f51a0 msgr2=0x7f45040f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:54.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.028+0000 7f4509bf8700 1 --2- 172.21.15.123:0/2519562486 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45040f51a0 0x7f45040f55c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f44f8009fb0 tx=0x7f44f8009b30).stop 2024-01-13T11:55:54.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.028+0000 7f4509bf8700 1 -- 172.21.15.123:0/2519562486 shutdown_connections 2024-01-13T11:55:54.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.028+0000 7f4509bf8700 1 --2- 172.21.15.123:0/2519562486 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45040f51a0 0x7f45040f55c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:54.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.028+0000 7f4509bf8700 1 -- 172.21.15.123:0/2519562486 >> 172.21.15.123:0/2519562486 conn(0x7f45040f09a0 msgr2=0x7f45040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:54.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.028+0000 7f4509bf8700 1 -- 172.21.15.123:0/2519562486 shutdown_connections 2024-01-13T11:55:54.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.028+0000 7f4509bf8700 1 -- 172.21.15.123:0/2519562486 wait complete. 2024-01-13T11:55:54.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.029+0000 7f4509bf8700 1 Processor -- start 2024-01-13T11:55:54.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.029+0000 7f4509bf8700 1 -- start start 2024-01-13T11:55:54.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.029+0000 7f4509bf8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45040f51a0 0x7f45041091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:54.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.029+0000 7f4509bf8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4504060240 con 0x7f45040f51a0 2024-01-13T11:55:54.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.029+0000 7f4508bf6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45040f51a0 0x7f45041091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:54.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.029+0000 7f4508bf6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45040f51a0 0x7f45041091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:53662/0 (socket says 172.21.15.123:53662) 2024-01-13T11:55:54.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.029+0000 7f4508bf6700 1 -- 172.21.15.123:0/1054236157 learned_addr learned my addr 172.21.15.123:0/1054236157 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:54.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.030+0000 7f4508bf6700 1 -- 172.21.15.123:0/1054236157 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f8005040 con 0x7f45040f51a0 2024-01-13T11:55:54.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.030+0000 7f4508bf6700 1 --2- 172.21.15.123:0/1054236157 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45040f51a0 0x7f45041091b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f44f8013510 tx=0x7f44f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:54.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.030+0000 7f45017fa700 1 -- 172.21.15.123:0/1054236157 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44f8009320 con 0x7f45040f51a0 2024-01-13T11:55:54.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.030+0000 7f4509bf8700 1 -- 172.21.15.123:0/1054236157 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4504060440 con 0x7f45040f51a0 2024-01-13T11:55:54.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.030+0000 7f45017fa700 1 -- 172.21.15.123:0/1054236157 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44f8013f00 con 0x7f45040f51a0 2024-01-13T11:55:54.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.030+0000 7f45017fa700 1 -- 172.21.15.123:0/1054236157 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44f8014dc0 con 0x7f45040f51a0 2024-01-13T11:55:54.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.030+0000 7f4509bf8700 1 -- 172.21.15.123:0/1054236157 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4504109860 con 0x7f45040f51a0 2024-01-13T11:55:54.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.031+0000 7f45017fa700 1 -- 172.21.15.123:0/1054236157 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44f80095f0 con 0x7f45040f51a0 2024-01-13T11:55:54.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.031+0000 7f4509bf8700 1 -- 172.21.15.123:0/1054236157 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4504045ad0 con 0x7f45040f51a0 2024-01-13T11:55:54.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.031+0000 7f45017fa700 1 --2- 172.21.15.123:0/1054236157 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f44f402ff60 0x7f44f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:54.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.031+0000 7f45017fa700 1 -- 172.21.15.123:0/1054236157 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f44f80410e0 con 0x7f45040f51a0 2024-01-13T11:55:54.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.031+0000 7f4503fff700 1 --2- 172.21.15.123:0/1054236157 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f44f402ff60 0x7f44f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:54.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.032+0000 7f4503fff700 1 --2- 172.21.15.123:0/1054236157 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f44f402ff60 0x7f44f4032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f44f00096a0 tx=0x7f44f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:54.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.036+0000 7f45017fa700 1 -- 172.21.15.123:0/1054236157 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44f8014420 con 0x7f45040f51a0 2024-01-13T11:55:54.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.238+0000 7f4509bf8700 1 -- 172.21.15.123:0/1054236157 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f450410a080 con 0x7f45040f51a0 2024-01-13T11:55:54.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.239+0000 7f45017fa700 1 -- 172.21.15.123:0/1054236157 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f44f801c920 con 0x7f45040f51a0 2024-01-13T11:55:54.227 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:54.227 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:54.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.241+0000 7f4509bf8700 1 -- 172.21.15.123:0/1054236157 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f44f402ff60 msgr2=0x7f44f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:54.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.241+0000 7f4509bf8700 1 --2- 172.21.15.123:0/1054236157 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f44f402ff60 0x7f44f4032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f44f00096a0 tx=0x7f44f0006b40).stop 2024-01-13T11:55:54.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.241+0000 7f4509bf8700 1 -- 172.21.15.123:0/1054236157 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45040f51a0 msgr2=0x7f45041091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:54.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.241+0000 7f4509bf8700 1 --2- 172.21.15.123:0/1054236157 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45040f51a0 0x7f45041091b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f44f8013510 tx=0x7f44f800b040).stop 2024-01-13T11:55:54.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.241+0000 7f4509bf8700 1 -- 172.21.15.123:0/1054236157 shutdown_connections 2024-01-13T11:55:54.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.241+0000 7f4509bf8700 1 --2- 172.21.15.123:0/1054236157 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f44f402ff60 0x7f44f4032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:54.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.241+0000 7f4509bf8700 1 --2- 172.21.15.123:0/1054236157 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45040f51a0 0x7f45041091b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:54.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.241+0000 7f4509bf8700 1 -- 172.21.15.123:0/1054236157 >> 172.21.15.123:0/1054236157 conn(0x7f45040f09a0 msgr2=0x7f45040f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:54.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.241+0000 7f4509bf8700 1 -- 172.21.15.123:0/1054236157 shutdown_connections 2024-01-13T11:55:54.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:54.241+0000 7f4509bf8700 1 -- 172.21.15.123:0/1054236157 wait complete. 2024-01-13T11:55:54.230 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:54.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:54 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:54.240366+0000 mon.smithi059 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.123:0/1054236157' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:55.615 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:55.616 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:55.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:55 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:53.559361+0000 mgr.smithi059.hmyldj (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:56.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.768+0000 7f5ae0eea700 1 -- 172.21.15.123:0/1399121148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc0f4740 msgr2=0x7f5adc0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:56.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.768+0000 7f5ae0eea700 1 --2- 172.21.15.123:0/1399121148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc0f4740 0x7f5adc0f4b60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f5ad0009fb0 tx=0x7f5ad0009b30).stop 2024-01-13T11:55:56.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.769+0000 7f5ae0eea700 1 -- 172.21.15.123:0/1399121148 shutdown_connections 2024-01-13T11:55:56.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.769+0000 7f5ae0eea700 1 --2- 172.21.15.123:0/1399121148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc0f4740 0x7f5adc0f4b60 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:56.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.769+0000 7f5ae0eea700 1 -- 172.21.15.123:0/1399121148 >> 172.21.15.123:0/1399121148 conn(0x7f5adc0f0260 msgr2=0x7f5adc0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:56.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.769+0000 7f5ae0eea700 1 -- 172.21.15.123:0/1399121148 shutdown_connections 2024-01-13T11:55:56.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.769+0000 7f5ae0eea700 1 -- 172.21.15.123:0/1399121148 wait complete. 2024-01-13T11:55:56.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.769+0000 7f5ae0eea700 1 Processor -- start 2024-01-13T11:55:56.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.769+0000 7f5ae0eea700 1 -- start start 2024-01-13T11:55:56.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.769+0000 7f5ae0eea700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc0f4740 0x7f5adc0f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:56.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.769+0000 7f5ae0eea700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc0f5770 con 0x7f5adc0f4740 2024-01-13T11:55:56.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.770+0000 7f5ada59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc0f4740 0x7f5adc0f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:56.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.770+0000 7f5ada59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc0f4740 0x7f5adc0f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:53320/0 (socket says 172.21.15.123:53320) 2024-01-13T11:55:56.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.770+0000 7f5ada59c700 1 -- 172.21.15.123:0/3496746183 learned_addr learned my addr 172.21.15.123:0/3496746183 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:56.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.770+0000 7f5ada59c700 1 -- 172.21.15.123:0/3496746183 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad0005040 con 0x7f5adc0f4740 2024-01-13T11:55:56.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.770+0000 7f5ada59c700 1 --2- 172.21.15.123:0/3496746183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc0f4740 0x7f5adc0f72f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f5ad0013510 tx=0x7f5ad000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:56.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.770+0000 7f5acaffd700 1 -- 172.21.15.123:0/3496746183 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ad0009320 con 0x7f5adc0f4740 2024-01-13T11:55:56.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.770+0000 7f5ae0eea700 1 -- 172.21.15.123:0/3496746183 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5adc0f5910 con 0x7f5adc0f4740 2024-01-13T11:55:56.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.770+0000 7f5acaffd700 1 -- 172.21.15.123:0/3496746183 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ad0013f00 con 0x7f5adc0f4740 2024-01-13T11:55:56.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.770+0000 7f5acaffd700 1 -- 172.21.15.123:0/3496746183 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ad0014d90 con 0x7f5adc0f4740 2024-01-13T11:55:56.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.770+0000 7f5ae0eea700 1 -- 172.21.15.123:0/3496746183 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5adc0f5da0 con 0x7f5adc0f4740 2024-01-13T11:55:56.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.771+0000 7f5acaffd700 1 -- 172.21.15.123:0/3496746183 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ad00095f0 con 0x7f5adc0f4740 2024-01-13T11:55:56.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.771+0000 7f5ae0eea700 1 -- 172.21.15.123:0/3496746183 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5adc059100 con 0x7f5adc0f4740 2024-01-13T11:55:56.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.771+0000 7f5acaffd700 1 --2- 172.21.15.123:0/3496746183 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5ac402ff60 0x7f5ac4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:56.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.771+0000 7f5acaffd700 1 -- 172.21.15.123:0/3496746183 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ad0040fe0 con 0x7f5adc0f4740 2024-01-13T11:55:56.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.772+0000 7f5ad9d9b700 1 --2- 172.21.15.123:0/3496746183 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5ac402ff60 0x7f5ac4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:56.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.772+0000 7f5ad9d9b700 1 --2- 172.21.15.123:0/3496746183 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5ac402ff60 0x7f5ac4032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f5acc0096a0 tx=0x7f5acc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:56.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.776+0000 7f5acaffd700 1 -- 172.21.15.123:0/3496746183 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ad001d7f0 con 0x7f5adc0f4740 2024-01-13T11:55:56.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.982+0000 7f5ae0eea700 1 -- 172.21.15.123:0/3496746183 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5adc0fbf40 con 0x7f5adc0f4740 2024-01-13T11:55:56.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.983+0000 7f5acaffd700 1 -- 172.21.15.123:0/3496746183 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ad0010230 con 0x7f5adc0f4740 2024-01-13T11:55:56.971 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:56.971 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:56.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.985+0000 7f5ae0eea700 1 -- 172.21.15.123:0/3496746183 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5ac402ff60 msgr2=0x7f5ac4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:56.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.985+0000 7f5ae0eea700 1 --2- 172.21.15.123:0/3496746183 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5ac402ff60 0x7f5ac4032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f5acc0096a0 tx=0x7f5acc006b40).stop 2024-01-13T11:55:56.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.985+0000 7f5ae0eea700 1 -- 172.21.15.123:0/3496746183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc0f4740 msgr2=0x7f5adc0f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:56.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.985+0000 7f5ae0eea700 1 --2- 172.21.15.123:0/3496746183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc0f4740 0x7f5adc0f72f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f5ad0013510 tx=0x7f5ad000b040).stop 2024-01-13T11:55:56.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.985+0000 7f5ae0eea700 1 -- 172.21.15.123:0/3496746183 shutdown_connections 2024-01-13T11:55:56.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.985+0000 7f5ae0eea700 1 --2- 172.21.15.123:0/3496746183 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f5ac402ff60 0x7f5ac4032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:56.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.985+0000 7f5ae0eea700 1 --2- 172.21.15.123:0/3496746183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc0f4740 0x7f5adc0f72f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:56.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.985+0000 7f5ae0eea700 1 -- 172.21.15.123:0/3496746183 >> 172.21.15.123:0/3496746183 conn(0x7f5adc0f0260 msgr2=0x7f5adc0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:56.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.985+0000 7f5ae0eea700 1 -- 172.21.15.123:0/3496746183 shutdown_connections 2024-01-13T11:55:56.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:56.985+0000 7f5ae0eea700 1 -- 172.21.15.123:0/3496746183 wait complete. 2024-01-13T11:55:56.974 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:55:57.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:55.559679+0000 mgr.smithi059.hmyldj (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:57.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:56.984505+0000 mon.smithi059 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.123:0/3496746183' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:55:58.351 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:55:58.351 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:55:59.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.463+0000 7f869f961700 1 -- 172.21.15.123:0/2222415087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8698060280 msgr2=0x7f86980f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:59.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.463+0000 7f869f961700 1 --2- 172.21.15.123:0/2222415087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8698060280 0x7f86980f90f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f8688009fb0 tx=0x7f8688009b30).stop 2024-01-13T11:55:59.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.463+0000 7f869f961700 1 -- 172.21.15.123:0/2222415087 shutdown_connections 2024-01-13T11:55:59.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.463+0000 7f869f961700 1 --2- 172.21.15.123:0/2222415087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8698060280 0x7f86980f90f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:59.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.463+0000 7f869f961700 1 -- 172.21.15.123:0/2222415087 >> 172.21.15.123:0/2222415087 conn(0x7f86980f0b70 msgr2=0x7f86980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:59.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.463+0000 7f869f961700 1 -- 172.21.15.123:0/2222415087 shutdown_connections 2024-01-13T11:55:59.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.463+0000 7f869f961700 1 -- 172.21.15.123:0/2222415087 wait complete. 2024-01-13T11:55:59.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.464+0000 7f869f961700 1 Processor -- start 2024-01-13T11:55:59.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.464+0000 7f869f961700 1 -- start start 2024-01-13T11:55:59.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.464+0000 7f869f961700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8698060280 0x7f86981066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:59.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.464+0000 7f869f961700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8698106bd0 con 0x7f8698060280 2024-01-13T11:55:59.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.464+0000 7f869d6fd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8698060280 0x7f86981066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:59.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.464+0000 7f869d6fd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8698060280 0x7f86981066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:53330/0 (socket says 172.21.15.123:53330) 2024-01-13T11:55:59.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.464+0000 7f869d6fd700 1 -- 172.21.15.123:0/1137431838 learned_addr learned my addr 172.21.15.123:0/1137431838 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:55:59.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.464+0000 7f869d6fd700 1 -- 172.21.15.123:0/1137431838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8688005040 con 0x7f8698060280 2024-01-13T11:55:59.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.464+0000 7f869d6fd700 1 --2- 172.21.15.123:0/1137431838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8698060280 0x7f86981066e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f8688013510 tx=0x7f868800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:59.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.464+0000 7f868dffb700 1 -- 172.21.15.123:0/1137431838 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8688009320 con 0x7f8698060280 2024-01-13T11:55:59.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.464+0000 7f869f961700 1 -- 172.21.15.123:0/1137431838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8698106dd0 con 0x7f8698060280 2024-01-13T11:55:59.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.465+0000 7f868dffb700 1 -- 172.21.15.123:0/1137431838 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8688013f00 con 0x7f8698060280 2024-01-13T11:55:59.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.465+0000 7f868dffb700 1 -- 172.21.15.123:0/1137431838 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8688014dc0 con 0x7f8698060280 2024-01-13T11:55:59.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.465+0000 7f869f961700 1 -- 172.21.15.123:0/1137431838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8698107260 con 0x7f8698060280 2024-01-13T11:55:59.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.465+0000 7f868dffb700 1 -- 172.21.15.123:0/1137431838 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86880095f0 con 0x7f8698060280 2024-01-13T11:55:59.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.466+0000 7f869f961700 1 -- 172.21.15.123:0/1137431838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8698045ad0 con 0x7f8698060280 2024-01-13T11:55:59.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.466+0000 7f868dffb700 1 --2- 172.21.15.123:0/1137431838 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f868402ff60 0x7f8684032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:55:59.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.466+0000 7f868dffb700 1 -- 172.21.15.123:0/1137431838 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f86880410e0 con 0x7f8698060280 2024-01-13T11:55:59.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.466+0000 7f869cefc700 1 --2- 172.21.15.123:0/1137431838 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f868402ff60 0x7f8684032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:55:59.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.466+0000 7f869cefc700 1 --2- 172.21.15.123:0/1137431838 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f868402ff60 0x7f8684032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f86940096a0 tx=0x7f8694006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:55:59.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.470+0000 7f868dffb700 1 -- 172.21.15.123:0/1137431838 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8688014420 con 0x7f8698060280 2024-01-13T11:55:59.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:55:59 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:57.559973+0000 mgr.smithi059.hmyldj (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:59.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.675+0000 7f869f961700 1 -- 172.21.15.123:0/1137431838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f86981080b0 con 0x7f8698060280 2024-01-13T11:55:59.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.676+0000 7f868dffb700 1 -- 172.21.15.123:0/1137431838 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f868801c920 con 0x7f8698060280 2024-01-13T11:55:59.664 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:55:59.664 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:55:59.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.678+0000 7f869f961700 1 -- 172.21.15.123:0/1137431838 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f868402ff60 msgr2=0x7f8684032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:59.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.678+0000 7f869f961700 1 --2- 172.21.15.123:0/1137431838 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f868402ff60 0x7f8684032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f86940096a0 tx=0x7f8694006b40).stop 2024-01-13T11:55:59.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.678+0000 7f869f961700 1 -- 172.21.15.123:0/1137431838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8698060280 msgr2=0x7f86981066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:55:59.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.678+0000 7f869f961700 1 --2- 172.21.15.123:0/1137431838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8698060280 0x7f86981066e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f8688013510 tx=0x7f868800b040).stop 2024-01-13T11:55:59.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.678+0000 7f869f961700 1 -- 172.21.15.123:0/1137431838 shutdown_connections 2024-01-13T11:55:59.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.678+0000 7f869f961700 1 --2- 172.21.15.123:0/1137431838 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f868402ff60 0x7f8684032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:59.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.678+0000 7f869f961700 1 --2- 172.21.15.123:0/1137431838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8698060280 0x7f86981066e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:55:59.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.678+0000 7f869f961700 1 -- 172.21.15.123:0/1137431838 >> 172.21.15.123:0/1137431838 conn(0x7f86980f0b70 msgr2=0x7f86980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:55:59.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.678+0000 7f869f961700 1 -- 172.21.15.123:0/1137431838 shutdown_connections 2024-01-13T11:55:59.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:55:59.678+0000 7f869f961700 1 -- 172.21.15.123:0/1137431838 wait complete. 2024-01-13T11:55:59.668 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:00.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:00 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:55:59.677541+0000 mon.smithi059 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.123:0/1137431838' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:01.045 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:01.045 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:01.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:01 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:55:59.560325+0000 mgr.smithi059.hmyldj (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:02.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.135+0000 7f8d4ccaa700 1 -- 172.21.15.123:0/2848092535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d480f78e0 msgr2=0x7f8d480f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:02.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.136+0000 7f8d4ccaa700 1 --2- 172.21.15.123:0/2848092535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d480f78e0 0x7f8d480f7d00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f8d38009fb0 tx=0x7f8d38009b30).stop 2024-01-13T11:56:02.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.136+0000 7f8d4ccaa700 1 -- 172.21.15.123:0/2848092535 shutdown_connections 2024-01-13T11:56:02.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.136+0000 7f8d4ccaa700 1 --2- 172.21.15.123:0/2848092535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d480f78e0 0x7f8d480f7d00 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:02.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.136+0000 7f8d4ccaa700 1 -- 172.21.15.123:0/2848092535 >> 172.21.15.123:0/2848092535 conn(0x7f8d480f0120 msgr2=0x7f8d480f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:02.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.136+0000 7f8d4ccaa700 1 -- 172.21.15.123:0/2848092535 shutdown_connections 2024-01-13T11:56:02.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.136+0000 7f8d4ccaa700 1 -- 172.21.15.123:0/2848092535 wait complete. 2024-01-13T11:56:02.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.136+0000 7f8d4ccaa700 1 Processor -- start 2024-01-13T11:56:02.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.136+0000 7f8d4ccaa700 1 -- start start 2024-01-13T11:56:02.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.136+0000 7f8d4ccaa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d480f78e0 0x7f8d481064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:02.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.136+0000 7f8d4ccaa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d481069d0 con 0x7f8d480f78e0 2024-01-13T11:56:02.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.137+0000 7f8d477fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d480f78e0 0x7f8d481064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:02.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.137+0000 7f8d477fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d480f78e0 0x7f8d481064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:53356/0 (socket says 172.21.15.123:53356) 2024-01-13T11:56:02.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.137+0000 7f8d477fe700 1 -- 172.21.15.123:0/3232180592 learned_addr learned my addr 172.21.15.123:0/3232180592 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:02.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.137+0000 7f8d477fe700 1 -- 172.21.15.123:0/3232180592 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d38005040 con 0x7f8d480f78e0 2024-01-13T11:56:02.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.137+0000 7f8d477fe700 1 --2- 172.21.15.123:0/3232180592 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d480f78e0 0x7f8d481064e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f8d38013510 tx=0x7f8d3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:02.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.137+0000 7f8d2ffff700 1 -- 172.21.15.123:0/3232180592 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d38009320 con 0x7f8d480f78e0 2024-01-13T11:56:02.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.137+0000 7f8d4ccaa700 1 -- 172.21.15.123:0/3232180592 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d48106bd0 con 0x7f8d480f78e0 2024-01-13T11:56:02.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.137+0000 7f8d2ffff700 1 -- 172.21.15.123:0/3232180592 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d38013f00 con 0x7f8d480f78e0 2024-01-13T11:56:02.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.137+0000 7f8d2ffff700 1 -- 172.21.15.123:0/3232180592 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d38014dc0 con 0x7f8d480f78e0 2024-01-13T11:56:02.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.137+0000 7f8d4ccaa700 1 -- 172.21.15.123:0/3232180592 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d48107060 con 0x7f8d480f78e0 2024-01-13T11:56:02.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.138+0000 7f8d2ffff700 1 -- 172.21.15.123:0/3232180592 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8d3801d3d0 con 0x7f8d480f78e0 2024-01-13T11:56:02.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.138+0000 7f8d2ffff700 1 --2- 172.21.15.123:0/3232180592 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8d3002ff60 0x7f8d30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:02.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.138+0000 7f8d2ffff700 1 -- 172.21.15.123:0/3232180592 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8d38041010 con 0x7f8d480f78e0 2024-01-13T11:56:02.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.138+0000 7f8d4ccaa700 1 -- 172.21.15.123:0/3232180592 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d48045ad0 con 0x7f8d480f78e0 2024-01-13T11:56:02.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.139+0000 7f8d46ffd700 1 --2- 172.21.15.123:0/3232180592 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8d3002ff60 0x7f8d30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:02.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.139+0000 7f8d46ffd700 1 --2- 172.21.15.123:0/3232180592 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8d3002ff60 0x7f8d30032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f8d3c0096a0 tx=0x7f8d3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:02.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.143+0000 7f8d2ffff700 1 -- 172.21.15.123:0/3232180592 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d38012900 con 0x7f8d480f78e0 2024-01-13T11:56:02.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.351+0000 7f8d4ccaa700 1 -- 172.21.15.123:0/3232180592 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d48107d80 con 0x7f8d480f78e0 2024-01-13T11:56:02.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.352+0000 7f8d2ffff700 1 -- 172.21.15.123:0/3232180592 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8d3802b020 con 0x7f8d480f78e0 2024-01-13T11:56:02.340 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:02.340 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:02.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.354+0000 7f8d4ccaa700 1 -- 172.21.15.123:0/3232180592 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8d3002ff60 msgr2=0x7f8d30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:02.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.354+0000 7f8d4ccaa700 1 --2- 172.21.15.123:0/3232180592 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8d3002ff60 0x7f8d30032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f8d3c0096a0 tx=0x7f8d3c006b40).stop 2024-01-13T11:56:02.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.354+0000 7f8d4ccaa700 1 -- 172.21.15.123:0/3232180592 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d480f78e0 msgr2=0x7f8d481064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:02.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.354+0000 7f8d4ccaa700 1 --2- 172.21.15.123:0/3232180592 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d480f78e0 0x7f8d481064e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f8d38013510 tx=0x7f8d3800b040).stop 2024-01-13T11:56:02.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.354+0000 7f8d4ccaa700 1 -- 172.21.15.123:0/3232180592 shutdown_connections 2024-01-13T11:56:02.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.354+0000 7f8d4ccaa700 1 --2- 172.21.15.123:0/3232180592 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8d3002ff60 0x7f8d30032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:02.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.354+0000 7f8d4ccaa700 1 --2- 172.21.15.123:0/3232180592 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d480f78e0 0x7f8d481064e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:02.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.354+0000 7f8d4ccaa700 1 -- 172.21.15.123:0/3232180592 >> 172.21.15.123:0/3232180592 conn(0x7f8d480f0120 msgr2=0x7f8d480f0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:02.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.354+0000 7f8d4ccaa700 1 -- 172.21.15.123:0/3232180592 shutdown_connections 2024-01-13T11:56:02.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:02.354+0000 7f8d4ccaa700 1 -- 172.21.15.123:0/3232180592 wait complete. 2024-01-13T11:56:02.343 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:02.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:02 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:01.560681+0000 mgr.smithi059.hmyldj (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:03.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:03 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:02.353568+0000 mon.smithi059 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.123:0/3232180592' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:03.732 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:03.732 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:04.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:04 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:03.560957+0000 mgr.smithi059.hmyldj (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:04.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.794+0000 7ff7af622700 1 -- 172.21.15.123:0/2058265033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7a80f6510 msgr2=0x7ff7a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:04.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.794+0000 7ff7af622700 1 --2- 172.21.15.123:0/2058265033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7a80f6510 0x7ff7a80f6930 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7ff798009fb0 tx=0x7ff798009b30).stop 2024-01-13T11:56:04.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.794+0000 7ff7af622700 1 -- 172.21.15.123:0/2058265033 shutdown_connections 2024-01-13T11:56:04.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.794+0000 7ff7af622700 1 --2- 172.21.15.123:0/2058265033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7a80f6510 0x7ff7a80f6930 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:04.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.794+0000 7ff7af622700 1 -- 172.21.15.123:0/2058265033 >> 172.21.15.123:0/2058265033 conn(0x7ff7a80f1c70 msgr2=0x7ff7a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:04.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.794+0000 7ff7af622700 1 -- 172.21.15.123:0/2058265033 shutdown_connections 2024-01-13T11:56:04.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.794+0000 7ff7af622700 1 -- 172.21.15.123:0/2058265033 wait complete. 2024-01-13T11:56:04.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.795+0000 7ff7af622700 1 Processor -- start 2024-01-13T11:56:04.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.795+0000 7ff7af622700 1 -- start start 2024-01-13T11:56:04.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.795+0000 7ff7af622700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7a80f6510 0x7ff7a8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:04.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.795+0000 7ff7af622700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a8106c20 con 0x7ff7a80f6510 2024-01-13T11:56:04.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.795+0000 7ff7ad3be700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7a80f6510 0x7ff7a8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:04.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.795+0000 7ff7ad3be700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7a80f6510 0x7ff7a8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:53386/0 (socket says 172.21.15.123:53386) 2024-01-13T11:56:04.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.795+0000 7ff7ad3be700 1 -- 172.21.15.123:0/2482318723 learned_addr learned my addr 172.21.15.123:0/2482318723 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:04.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.796+0000 7ff7ad3be700 1 -- 172.21.15.123:0/2482318723 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff798005040 con 0x7ff7a80f6510 2024-01-13T11:56:04.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.796+0000 7ff7ad3be700 1 --2- 172.21.15.123:0/2482318723 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7a80f6510 0x7ff7a8106730 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7ff798013510 tx=0x7ff79800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:04.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.796+0000 7ff79dffb700 1 -- 172.21.15.123:0/2482318723 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff798009320 con 0x7ff7a80f6510 2024-01-13T11:56:04.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.796+0000 7ff79dffb700 1 -- 172.21.15.123:0/2482318723 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff798013f00 con 0x7ff7a80f6510 2024-01-13T11:56:04.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.796+0000 7ff79dffb700 1 -- 172.21.15.123:0/2482318723 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff798014d90 con 0x7ff7a80f6510 2024-01-13T11:56:04.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.796+0000 7ff7af622700 1 -- 172.21.15.123:0/2482318723 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a8106e20 con 0x7ff7a80f6510 2024-01-13T11:56:04.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.796+0000 7ff7af622700 1 -- 172.21.15.123:0/2482318723 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a81072b0 con 0x7ff7a80f6510 2024-01-13T11:56:04.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.797+0000 7ff7af622700 1 -- 172.21.15.123:0/2482318723 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7a8045ad0 con 0x7ff7a80f6510 2024-01-13T11:56:04.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.797+0000 7ff79dffb700 1 -- 172.21.15.123:0/2482318723 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7980095f0 con 0x7ff7a80f6510 2024-01-13T11:56:04.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.797+0000 7ff79dffb700 1 --2- 172.21.15.123:0/2482318723 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff79402ff60 0x7ff794032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:04.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.797+0000 7ff79dffb700 1 -- 172.21.15.123:0/2482318723 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff798041080 con 0x7ff7a80f6510 2024-01-13T11:56:04.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.798+0000 7ff7acbbd700 1 --2- 172.21.15.123:0/2482318723 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff79402ff60 0x7ff794032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:04.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.798+0000 7ff7acbbd700 1 --2- 172.21.15.123:0/2482318723 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff79402ff60 0x7ff794032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7ff7a40096a0 tx=0x7ff7a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:04.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:04.803+0000 7ff79dffb700 1 -- 172.21.15.123:0/2482318723 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff798014420 con 0x7ff7a80f6510 2024-01-13T11:56:04.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:05.010+0000 7ff7af622700 1 -- 172.21.15.123:0/2482318723 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7a80fbf60 con 0x7ff7a80f6510 2024-01-13T11:56:04.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:05.011+0000 7ff79dffb700 1 -- 172.21.15.123:0/2482318723 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff798010230 con 0x7ff7a80f6510 2024-01-13T11:56:04.999 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:04.999 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:05.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:05.014+0000 7ff7af622700 1 -- 172.21.15.123:0/2482318723 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff79402ff60 msgr2=0x7ff794032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:05.014+0000 7ff7af622700 1 --2- 172.21.15.123:0/2482318723 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff79402ff60 0x7ff794032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7ff7a40096a0 tx=0x7ff7a4006b40).stop 2024-01-13T11:56:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:05.014+0000 7ff7af622700 1 -- 172.21.15.123:0/2482318723 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7a80f6510 msgr2=0x7ff7a8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:05.014+0000 7ff7af622700 1 --2- 172.21.15.123:0/2482318723 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7a80f6510 0x7ff7a8106730 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7ff798013510 tx=0x7ff79800b040).stop 2024-01-13T11:56:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:05.014+0000 7ff7af622700 1 -- 172.21.15.123:0/2482318723 shutdown_connections 2024-01-13T11:56:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:05.014+0000 7ff7af622700 1 --2- 172.21.15.123:0/2482318723 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff79402ff60 0x7ff794032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:05.014+0000 7ff7af622700 1 --2- 172.21.15.123:0/2482318723 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7a80f6510 0x7ff7a8106730 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:05.014+0000 7ff7af622700 1 -- 172.21.15.123:0/2482318723 >> 172.21.15.123:0/2482318723 conn(0x7ff7a80f1c70 msgr2=0x7ff7a80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:05.014+0000 7ff7af622700 1 -- 172.21.15.123:0/2482318723 shutdown_connections 2024-01-13T11:56:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:05.014+0000 7ff7af622700 1 -- 172.21.15.123:0/2482318723 wait complete. 2024-01-13T11:56:05.001 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:05.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:05 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:05.012506+0000 mon.smithi059 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.123:0/2482318723' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:06.416 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:06.416 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:07.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.617+0000 7ff46d5e4700 1 -- 172.21.15.123:0/1404943357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4680f47c0 msgr2=0x7ff4680f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:07.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.617+0000 7ff46d5e4700 1 --2- 172.21.15.123:0/1404943357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4680f47c0 0x7ff4680f4be0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7ff45c009fb0 tx=0x7ff45c009b30).stop 2024-01-13T11:56:07.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.617+0000 7ff46d5e4700 1 -- 172.21.15.123:0/1404943357 shutdown_connections 2024-01-13T11:56:07.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.617+0000 7ff46d5e4700 1 --2- 172.21.15.123:0/1404943357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4680f47c0 0x7ff4680f4be0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:07.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.617+0000 7ff46d5e4700 1 -- 172.21.15.123:0/1404943357 >> 172.21.15.123:0/1404943357 conn(0x7ff4680f02e0 msgr2=0x7ff4680f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:07.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.617+0000 7ff46d5e4700 1 -- 172.21.15.123:0/1404943357 shutdown_connections 2024-01-13T11:56:07.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.617+0000 7ff46d5e4700 1 -- 172.21.15.123:0/1404943357 wait complete. 2024-01-13T11:56:07.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.618+0000 7ff46d5e4700 1 Processor -- start 2024-01-13T11:56:07.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.618+0000 7ff46d5e4700 1 -- start start 2024-01-13T11:56:07.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.618+0000 7ff46d5e4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4680f47c0 0x7ff468106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:07.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.618+0000 7ff46d5e4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff468106bf0 con 0x7ff4680f47c0 2024-01-13T11:56:07.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.618+0000 7ff466ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4680f47c0 0x7ff468106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:07.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.618+0000 7ff466ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4680f47c0 0x7ff468106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:39910/0 (socket says 172.21.15.123:39910) 2024-01-13T11:56:07.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.618+0000 7ff466ffd700 1 -- 172.21.15.123:0/903115545 learned_addr learned my addr 172.21.15.123:0/903115545 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:07.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.619+0000 7ff466ffd700 1 -- 172.21.15.123:0/903115545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45c005040 con 0x7ff4680f47c0 2024-01-13T11:56:07.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.619+0000 7ff466ffd700 1 --2- 172.21.15.123:0/903115545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4680f47c0 0x7ff468106700 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7ff45c013510 tx=0x7ff45c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:07.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.619+0000 7ff44f7fe700 1 -- 172.21.15.123:0/903115545 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff45c009320 con 0x7ff4680f47c0 2024-01-13T11:56:07.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.619+0000 7ff46d5e4700 1 -- 172.21.15.123:0/903115545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff468106df0 con 0x7ff4680f47c0 2024-01-13T11:56:07.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.619+0000 7ff44f7fe700 1 -- 172.21.15.123:0/903115545 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff45c013f00 con 0x7ff4680f47c0 2024-01-13T11:56:07.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.619+0000 7ff44f7fe700 1 -- 172.21.15.123:0/903115545 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff45c014d90 con 0x7ff4680f47c0 2024-01-13T11:56:07.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.619+0000 7ff46d5e4700 1 -- 172.21.15.123:0/903115545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff468107280 con 0x7ff4680f47c0 2024-01-13T11:56:07.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.620+0000 7ff44f7fe700 1 -- 172.21.15.123:0/903115545 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff45c014420 con 0x7ff4680f47c0 2024-01-13T11:56:07.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.620+0000 7ff46d5e4700 1 -- 172.21.15.123:0/903115545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff468045ad0 con 0x7ff4680f47c0 2024-01-13T11:56:07.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.620+0000 7ff44f7fe700 1 --2- 172.21.15.123:0/903115545 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff45002ff60 0x7ff450032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:07.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.620+0000 7ff44f7fe700 1 -- 172.21.15.123:0/903115545 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff45c041110 con 0x7ff4680f47c0 2024-01-13T11:56:07.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.620+0000 7ff4667fc700 1 --2- 172.21.15.123:0/903115545 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff45002ff60 0x7ff450032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:07.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.621+0000 7ff4667fc700 1 --2- 172.21.15.123:0/903115545 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff45002ff60 0x7ff450032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7ff4580096a0 tx=0x7ff458006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:07.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.625+0000 7ff44f7fe700 1 -- 172.21.15.123:0/903115545 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff45c0095f0 con 0x7ff4680f47c0 2024-01-13T11:56:07.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:05.561256+0000 mgr.smithi059.hmyldj (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:07.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.833+0000 7ff46d5e4700 1 -- 172.21.15.123:0/903115545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4681080d0 con 0x7ff4680f47c0 2024-01-13T11:56:07.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.833+0000 7ff44f7fe700 1 -- 172.21.15.123:0/903115545 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff45c01db70 con 0x7ff4680f47c0 2024-01-13T11:56:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:07.821 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:07.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.835+0000 7ff46d5e4700 1 -- 172.21.15.123:0/903115545 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff45002ff60 msgr2=0x7ff450032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:07.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.835+0000 7ff46d5e4700 1 --2- 172.21.15.123:0/903115545 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff45002ff60 0x7ff450032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7ff4580096a0 tx=0x7ff458006b40).stop 2024-01-13T11:56:07.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.835+0000 7ff46d5e4700 1 -- 172.21.15.123:0/903115545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4680f47c0 msgr2=0x7ff468106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:07.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.835+0000 7ff46d5e4700 1 --2- 172.21.15.123:0/903115545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4680f47c0 0x7ff468106700 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7ff45c013510 tx=0x7ff45c00b040).stop 2024-01-13T11:56:07.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.835+0000 7ff46d5e4700 1 -- 172.21.15.123:0/903115545 shutdown_connections 2024-01-13T11:56:07.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.835+0000 7ff46d5e4700 1 --2- 172.21.15.123:0/903115545 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff45002ff60 0x7ff450032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:07.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.835+0000 7ff46d5e4700 1 --2- 172.21.15.123:0/903115545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4680f47c0 0x7ff468106700 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:07.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.835+0000 7ff46d5e4700 1 -- 172.21.15.123:0/903115545 >> 172.21.15.123:0/903115545 conn(0x7ff4680f02e0 msgr2=0x7ff4680f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:07.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.835+0000 7ff46d5e4700 1 -- 172.21.15.123:0/903115545 shutdown_connections 2024-01-13T11:56:07.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:07.835+0000 7ff46d5e4700 1 -- 172.21.15.123:0/903115545 wait complete. 2024-01-13T11:56:07.825 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:08.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:08 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:07.833787+0000 mon.smithi059 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.123:0/903115545' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:09.211 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:09.211 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:09.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:09 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:07.561607+0000 mgr.smithi059.hmyldj (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:10.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.449+0000 7f47212f9700 1 -- 172.21.15.123:0/702892023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f471c060220 msgr2=0x7f471c0fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:10.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.449+0000 7f47212f9700 1 --2- 172.21.15.123:0/702892023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f471c060220 0x7f471c0fb550 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f470c009fb0 tx=0x7f470c009b30).stop 2024-01-13T11:56:10.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.449+0000 7f47212f9700 1 -- 172.21.15.123:0/702892023 shutdown_connections 2024-01-13T11:56:10.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.449+0000 7f47212f9700 1 --2- 172.21.15.123:0/702892023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f471c060220 0x7f471c0fb550 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:10.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.449+0000 7f47212f9700 1 -- 172.21.15.123:0/702892023 >> 172.21.15.123:0/702892023 conn(0x7f471c0f0b70 msgr2=0x7f471c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:10.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.449+0000 7f47212f9700 1 -- 172.21.15.123:0/702892023 shutdown_connections 2024-01-13T11:56:10.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.449+0000 7f47212f9700 1 -- 172.21.15.123:0/702892023 wait complete. 2024-01-13T11:56:10.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.450+0000 7f47212f9700 1 Processor -- start 2024-01-13T11:56:10.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.450+0000 7f47212f9700 1 -- start start 2024-01-13T11:56:10.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.450+0000 7f47212f9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f471c060220 0x7f471c1088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:10.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.450+0000 7f47212f9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c108da0 con 0x7f471c060220 2024-01-13T11:56:10.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.451+0000 7f471affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f471c060220 0x7f471c1088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:10.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.451+0000 7f471affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f471c060220 0x7f471c1088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:39938/0 (socket says 172.21.15.123:39938) 2024-01-13T11:56:10.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.451+0000 7f471affd700 1 -- 172.21.15.123:0/2042644537 learned_addr learned my addr 172.21.15.123:0/2042644537 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:10.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.451+0000 7f471affd700 1 -- 172.21.15.123:0/2042644537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470c005040 con 0x7f471c060220 2024-01-13T11:56:10.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.451+0000 7f471affd700 1 --2- 172.21.15.123:0/2042644537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f471c060220 0x7f471c1088b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f470c013510 tx=0x7f470c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:10.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.451+0000 7f47037fe700 1 -- 172.21.15.123:0/2042644537 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f470c009320 con 0x7f471c060220 2024-01-13T11:56:10.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.451+0000 7f47212f9700 1 -- 172.21.15.123:0/2042644537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f471c108fa0 con 0x7f471c060220 2024-01-13T11:56:10.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.451+0000 7f47037fe700 1 -- 172.21.15.123:0/2042644537 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f470c013f00 con 0x7f471c060220 2024-01-13T11:56:10.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.451+0000 7f47037fe700 1 -- 172.21.15.123:0/2042644537 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f470c014dc0 con 0x7f471c060220 2024-01-13T11:56:10.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.451+0000 7f47212f9700 1 -- 172.21.15.123:0/2042644537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471c109430 con 0x7f471c060220 2024-01-13T11:56:10.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.452+0000 7f47037fe700 1 -- 172.21.15.123:0/2042644537 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f470c0095f0 con 0x7f471c060220 2024-01-13T11:56:10.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.452+0000 7f47212f9700 1 -- 172.21.15.123:0/2042644537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471c045ad0 con 0x7f471c060220 2024-01-13T11:56:10.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.452+0000 7f47037fe700 1 --2- 172.21.15.123:0/2042644537 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f470402ff60 0x7f4704032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:10.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.452+0000 7f47037fe700 1 -- 172.21.15.123:0/2042644537 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f470c0410c0 con 0x7f471c060220 2024-01-13T11:56:10.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.453+0000 7f471a7fc700 1 --2- 172.21.15.123:0/2042644537 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f470402ff60 0x7f4704032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:10.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.453+0000 7f471a7fc700 1 --2- 172.21.15.123:0/2042644537 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f470402ff60 0x7f4704032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f47100096a0 tx=0x7f4710006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:10.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.457+0000 7f47037fe700 1 -- 172.21.15.123:0/2042644537 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f470c014420 con 0x7f471c060220 2024-01-13T11:56:10.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.660+0000 7f47212f9700 1 -- 172.21.15.123:0/2042644537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f471c10a2e0 con 0x7f471c060220 2024-01-13T11:56:10.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.661+0000 7f47037fe700 1 -- 172.21.15.123:0/2042644537 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f470c010230 con 0x7f471c060220 2024-01-13T11:56:10.649 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:10.649 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:10.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.662+0000 7f47212f9700 1 -- 172.21.15.123:0/2042644537 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f470402ff60 msgr2=0x7f4704032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:10.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.662+0000 7f47212f9700 1 --2- 172.21.15.123:0/2042644537 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f470402ff60 0x7f4704032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f47100096a0 tx=0x7f4710006b40).stop 2024-01-13T11:56:10.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.662+0000 7f47212f9700 1 -- 172.21.15.123:0/2042644537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f471c060220 msgr2=0x7f471c1088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:10.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.662+0000 7f47212f9700 1 --2- 172.21.15.123:0/2042644537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f471c060220 0x7f471c1088b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f470c013510 tx=0x7f470c00b040).stop 2024-01-13T11:56:10.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.663+0000 7f47212f9700 1 -- 172.21.15.123:0/2042644537 shutdown_connections 2024-01-13T11:56:10.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.663+0000 7f47212f9700 1 --2- 172.21.15.123:0/2042644537 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f470402ff60 0x7f4704032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:10.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.663+0000 7f47212f9700 1 --2- 172.21.15.123:0/2042644537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f471c060220 0x7f471c1088b0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:10.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.663+0000 7f47212f9700 1 -- 172.21.15.123:0/2042644537 >> 172.21.15.123:0/2042644537 conn(0x7f471c0f0b70 msgr2=0x7f471c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:10.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.663+0000 7f47212f9700 1 -- 172.21.15.123:0/2042644537 shutdown_connections 2024-01-13T11:56:10.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:10.663+0000 7f47212f9700 1 -- 172.21.15.123:0/2042644537 wait complete. 2024-01-13T11:56:10.651 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:11.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:11 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:09.561988+0000 mgr.smithi059.hmyldj (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:11.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:11 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:10.662219+0000 mon.smithi059 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.123:0/2042644537' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:12.029 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:12.030 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:13.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.149+0000 7f319c15a700 1 -- 172.21.15.123:0/47229474 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31940f5410 msgr2=0x7f31940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:13.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.149+0000 7f319c15a700 1 --2- 172.21.15.123:0/47229474 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31940f5410 0x7f31940f5830 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f3184009fb0 tx=0x7f3184009b30).stop 2024-01-13T11:56:13.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.149+0000 7f319c15a700 1 -- 172.21.15.123:0/47229474 shutdown_connections 2024-01-13T11:56:13.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.149+0000 7f319c15a700 1 --2- 172.21.15.123:0/47229474 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31940f5410 0x7f31940f5830 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:13.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.149+0000 7f319c15a700 1 -- 172.21.15.123:0/47229474 >> 172.21.15.123:0/47229474 conn(0x7f31940f0b70 msgr2=0x7f31940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:13.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.149+0000 7f319c15a700 1 -- 172.21.15.123:0/47229474 shutdown_connections 2024-01-13T11:56:13.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.149+0000 7f319c15a700 1 -- 172.21.15.123:0/47229474 wait complete. 2024-01-13T11:56:13.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.150+0000 7f319c15a700 1 Processor -- start 2024-01-13T11:56:13.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.150+0000 7f319c15a700 1 -- start start 2024-01-13T11:56:13.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.150+0000 7f319c15a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31940f5410 0x7f3194106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:13.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.150+0000 7f319c15a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3194106c10 con 0x7f31940f5410 2024-01-13T11:56:13.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.150+0000 7f3199ef6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31940f5410 0x7f3194106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:13.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.150+0000 7f3199ef6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31940f5410 0x7f3194106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:39956/0 (socket says 172.21.15.123:39956) 2024-01-13T11:56:13.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.150+0000 7f3199ef6700 1 -- 172.21.15.123:0/779147288 learned_addr learned my addr 172.21.15.123:0/779147288 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:13.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.150+0000 7f3199ef6700 1 -- 172.21.15.123:0/779147288 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3184005040 con 0x7f31940f5410 2024-01-13T11:56:13.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.150+0000 7f3199ef6700 1 --2- 172.21.15.123:0/779147288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31940f5410 0x7f3194106720 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f3184013510 tx=0x7f318400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:13.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.151+0000 7f318a7fc700 1 -- 172.21.15.123:0/779147288 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3184009320 con 0x7f31940f5410 2024-01-13T11:56:13.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.151+0000 7f319c15a700 1 -- 172.21.15.123:0/779147288 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3194106e10 con 0x7f31940f5410 2024-01-13T11:56:13.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.151+0000 7f318a7fc700 1 -- 172.21.15.123:0/779147288 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3184013f00 con 0x7f31940f5410 2024-01-13T11:56:13.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.151+0000 7f318a7fc700 1 -- 172.21.15.123:0/779147288 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3184014d90 con 0x7f31940f5410 2024-01-13T11:56:13.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.151+0000 7f319c15a700 1 -- 172.21.15.123:0/779147288 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31941072a0 con 0x7f31940f5410 2024-01-13T11:56:13.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.152+0000 7f318a7fc700 1 -- 172.21.15.123:0/779147288 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f31840095f0 con 0x7f31940f5410 2024-01-13T11:56:13.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.151+0000 7f319c15a700 1 -- 172.21.15.123:0/779147288 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3194045ad0 con 0x7f31940f5410 2024-01-13T11:56:13.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.152+0000 7f318a7fc700 1 --2- 172.21.15.123:0/779147288 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f318002ff60 0x7f3180032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:13.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.152+0000 7f318a7fc700 1 -- 172.21.15.123:0/779147288 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3184041080 con 0x7f31940f5410 2024-01-13T11:56:13.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.152+0000 7f31996f5700 1 --2- 172.21.15.123:0/779147288 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f318002ff60 0x7f3180032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:13.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.152+0000 7f31996f5700 1 --2- 172.21.15.123:0/779147288 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f318002ff60 0x7f3180032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f31900096a0 tx=0x7f3190006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:13.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.156+0000 7f318a7fc700 1 -- 172.21.15.123:0/779147288 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3184014420 con 0x7f31940f5410 2024-01-13T11:56:13.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.363+0000 7f319c15a700 1 -- 172.21.15.123:0/779147288 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f31941080f0 con 0x7f31940f5410 2024-01-13T11:56:13.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.364+0000 7f318a7fc700 1 -- 172.21.15.123:0/779147288 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3184010230 con 0x7f31940f5410 2024-01-13T11:56:13.352 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:13.352 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:13.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.366+0000 7f319c15a700 1 -- 172.21.15.123:0/779147288 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f318002ff60 msgr2=0x7f3180032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:13.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.366+0000 7f319c15a700 1 --2- 172.21.15.123:0/779147288 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f318002ff60 0x7f3180032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f31900096a0 tx=0x7f3190006b40).stop 2024-01-13T11:56:13.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.366+0000 7f319c15a700 1 -- 172.21.15.123:0/779147288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31940f5410 msgr2=0x7f3194106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:13.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.366+0000 7f319c15a700 1 --2- 172.21.15.123:0/779147288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31940f5410 0x7f3194106720 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f3184013510 tx=0x7f318400b040).stop 2024-01-13T11:56:13.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.366+0000 7f319c15a700 1 -- 172.21.15.123:0/779147288 shutdown_connections 2024-01-13T11:56:13.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.366+0000 7f319c15a700 1 --2- 172.21.15.123:0/779147288 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f318002ff60 0x7f3180032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:13.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.366+0000 7f319c15a700 1 --2- 172.21.15.123:0/779147288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31940f5410 0x7f3194106720 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:13.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.366+0000 7f319c15a700 1 -- 172.21.15.123:0/779147288 >> 172.21.15.123:0/779147288 conn(0x7f31940f0b70 msgr2=0x7f319405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:13.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.366+0000 7f319c15a700 1 -- 172.21.15.123:0/779147288 shutdown_connections 2024-01-13T11:56:13.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:13.366+0000 7f319c15a700 1 -- 172.21.15.123:0/779147288 wait complete. 2024-01-13T11:56:13.355 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:13.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:13 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:11.562321+0000 mgr.smithi059.hmyldj (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:14.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:13.365376+0000 mon.smithi059 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.123:0/779147288' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:14.754 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:14.755 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:15.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:15 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:13.562701+0000 mgr.smithi059.hmyldj (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:15.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.899+0000 7fbded3e9700 1 -- 172.21.15.123:0/1586588647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbde80f5410 msgr2=0x7fbde80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:15.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.899+0000 7fbded3e9700 1 --2- 172.21.15.123:0/1586588647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbde80f5410 0x7fbde80f5830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fbddc009fb0 tx=0x7fbddc009b30).stop 2024-01-13T11:56:15.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.899+0000 7fbded3e9700 1 -- 172.21.15.123:0/1586588647 shutdown_connections 2024-01-13T11:56:15.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.899+0000 7fbded3e9700 1 --2- 172.21.15.123:0/1586588647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbde80f5410 0x7fbde80f5830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:15.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.899+0000 7fbded3e9700 1 -- 172.21.15.123:0/1586588647 >> 172.21.15.123:0/1586588647 conn(0x7fbde80f0b70 msgr2=0x7fbde80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:15.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.899+0000 7fbded3e9700 1 -- 172.21.15.123:0/1586588647 shutdown_connections 2024-01-13T11:56:15.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.899+0000 7fbded3e9700 1 -- 172.21.15.123:0/1586588647 wait complete. 2024-01-13T11:56:15.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.900+0000 7fbded3e9700 1 Processor -- start 2024-01-13T11:56:15.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.900+0000 7fbded3e9700 1 -- start start 2024-01-13T11:56:15.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.900+0000 7fbded3e9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbde80f5410 0x7fbde8109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:15.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.900+0000 7fbded3e9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde8060240 con 0x7fbde80f5410 2024-01-13T11:56:15.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.900+0000 7fbde6ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbde80f5410 0x7fbde8109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:15.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.900+0000 7fbde6ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbde80f5410 0x7fbde8109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:39974/0 (socket says 172.21.15.123:39974) 2024-01-13T11:56:15.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.900+0000 7fbde6ffd700 1 -- 172.21.15.123:0/587948619 learned_addr learned my addr 172.21.15.123:0/587948619 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:15.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.901+0000 7fbde6ffd700 1 -- 172.21.15.123:0/587948619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbddc005040 con 0x7fbde80f5410 2024-01-13T11:56:15.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.901+0000 7fbde6ffd700 1 --2- 172.21.15.123:0/587948619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbde80f5410 0x7fbde8109400 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fbddc013510 tx=0x7fbddc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:15.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.901+0000 7fbdcf7fe700 1 -- 172.21.15.123:0/587948619 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbddc009320 con 0x7fbde80f5410 2024-01-13T11:56:15.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.901+0000 7fbded3e9700 1 -- 172.21.15.123:0/587948619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde8060440 con 0x7fbde80f5410 2024-01-13T11:56:15.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.901+0000 7fbdcf7fe700 1 -- 172.21.15.123:0/587948619 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbddc013f00 con 0x7fbde80f5410 2024-01-13T11:56:15.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.901+0000 7fbdcf7fe700 1 -- 172.21.15.123:0/587948619 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbddc014d90 con 0x7fbde80f5410 2024-01-13T11:56:15.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.901+0000 7fbded3e9700 1 -- 172.21.15.123:0/587948619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde8109ab0 con 0x7fbde80f5410 2024-01-13T11:56:15.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.902+0000 7fbdcf7fe700 1 -- 172.21.15.123:0/587948619 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbddc0095f0 con 0x7fbde80f5410 2024-01-13T11:56:15.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.902+0000 7fbded3e9700 1 -- 172.21.15.123:0/587948619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde8045ad0 con 0x7fbde80f5410 2024-01-13T11:56:15.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.902+0000 7fbdcf7fe700 1 --2- 172.21.15.123:0/587948619 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbdd002ff60 0x7fbdd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:15.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.902+0000 7fbdcf7fe700 1 -- 172.21.15.123:0/587948619 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbddc041000 con 0x7fbde80f5410 2024-01-13T11:56:15.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.902+0000 7fbde67fc700 1 --2- 172.21.15.123:0/587948619 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbdd002ff60 0x7fbdd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:15.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.903+0000 7fbde67fc700 1 --2- 172.21.15.123:0/587948619 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbdd002ff60 0x7fbdd0032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fbdd80096a0 tx=0x7fbdd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:15.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:15.907+0000 7fbdcf7fe700 1 -- 172.21.15.123:0/587948619 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbddc014420 con 0x7fbde80f5410 2024-01-13T11:56:16.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:16.117+0000 7fbded3e9700 1 -- 172.21.15.123:0/587948619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbde80fae60 con 0x7fbde80f5410 2024-01-13T11:56:16.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:16.117+0000 7fbdcf7fe700 1 -- 172.21.15.123:0/587948619 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbddc010230 con 0x7fbde80f5410 2024-01-13T11:56:16.105 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:16.105 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:16.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:16.119+0000 7fbded3e9700 1 -- 172.21.15.123:0/587948619 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbdd002ff60 msgr2=0x7fbdd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:16.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:16.119+0000 7fbded3e9700 1 --2- 172.21.15.123:0/587948619 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbdd002ff60 0x7fbdd0032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fbdd80096a0 tx=0x7fbdd8006b40).stop 2024-01-13T11:56:16.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:16.119+0000 7fbded3e9700 1 -- 172.21.15.123:0/587948619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbde80f5410 msgr2=0x7fbde8109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:16.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:16.119+0000 7fbded3e9700 1 --2- 172.21.15.123:0/587948619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbde80f5410 0x7fbde8109400 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fbddc013510 tx=0x7fbddc00b040).stop 2024-01-13T11:56:16.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:16.119+0000 7fbded3e9700 1 -- 172.21.15.123:0/587948619 shutdown_connections 2024-01-13T11:56:16.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:16.119+0000 7fbded3e9700 1 --2- 172.21.15.123:0/587948619 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fbdd002ff60 0x7fbdd0032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:16.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:16.119+0000 7fbded3e9700 1 --2- 172.21.15.123:0/587948619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbde80f5410 0x7fbde8109400 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:16.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:16.119+0000 7fbded3e9700 1 -- 172.21.15.123:0/587948619 >> 172.21.15.123:0/587948619 conn(0x7fbde80f0b70 msgr2=0x7fbde80f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:16.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:16.119+0000 7fbded3e9700 1 -- 172.21.15.123:0/587948619 shutdown_connections 2024-01-13T11:56:16.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:16.119+0000 7fbded3e9700 1 -- 172.21.15.123:0/587948619 wait complete. 2024-01-13T11:56:16.108 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:16.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:16 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:16.118823+0000 mon.smithi059 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.123:0/587948619' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:17.477 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:17.478 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:17.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:17 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:15.563049+0000 mgr.smithi059.hmyldj (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:18.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.660+0000 7fdcadc41700 1 -- 172.21.15.123:0/2906706561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdca8060220 msgr2=0x7fdca80f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:18.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.660+0000 7fdcadc41700 1 --2- 172.21.15.123:0/2906706561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdca8060220 0x7fdca80f9150 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fdc98009fb0 tx=0x7fdc98009b30).stop 2024-01-13T11:56:18.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.660+0000 7fdcadc41700 1 -- 172.21.15.123:0/2906706561 shutdown_connections 2024-01-13T11:56:18.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.660+0000 7fdcadc41700 1 --2- 172.21.15.123:0/2906706561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdca8060220 0x7fdca80f9150 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:18.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.660+0000 7fdcadc41700 1 -- 172.21.15.123:0/2906706561 >> 172.21.15.123:0/2906706561 conn(0x7fdca80f0b70 msgr2=0x7fdca80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:18.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.660+0000 7fdcadc41700 1 -- 172.21.15.123:0/2906706561 shutdown_connections 2024-01-13T11:56:18.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.660+0000 7fdcadc41700 1 -- 172.21.15.123:0/2906706561 wait complete. 2024-01-13T11:56:18.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.661+0000 7fdcadc41700 1 Processor -- start 2024-01-13T11:56:18.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.661+0000 7fdcadc41700 1 -- start start 2024-01-13T11:56:18.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.661+0000 7fdcadc41700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdca8060220 0x7fdca80f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:18.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.661+0000 7fdcadc41700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca80f5050 con 0x7fdca8060220 2024-01-13T11:56:18.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.662+0000 7fdca77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdca8060220 0x7fdca80f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:18.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.662+0000 7fdca77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdca8060220 0x7fdca80f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:48772/0 (socket says 172.21.15.123:48772) 2024-01-13T11:56:18.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.662+0000 7fdca77fe700 1 -- 172.21.15.123:0/4283733349 learned_addr learned my addr 172.21.15.123:0/4283733349 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:18.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.662+0000 7fdca77fe700 1 -- 172.21.15.123:0/4283733349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc98005040 con 0x7fdca8060220 2024-01-13T11:56:18.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.662+0000 7fdca77fe700 1 --2- 172.21.15.123:0/4283733349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdca8060220 0x7fdca80f6bd0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fdc98013510 tx=0x7fdc9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:18.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.662+0000 7fdc8ffff700 1 -- 172.21.15.123:0/4283733349 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc98009320 con 0x7fdca8060220 2024-01-13T11:56:18.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.662+0000 7fdcadc41700 1 -- 172.21.15.123:0/4283733349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca80f51f0 con 0x7fdca8060220 2024-01-13T11:56:18.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.662+0000 7fdc8ffff700 1 -- 172.21.15.123:0/4283733349 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc98013f00 con 0x7fdca8060220 2024-01-13T11:56:18.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.662+0000 7fdc8ffff700 1 -- 172.21.15.123:0/4283733349 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc98014dc0 con 0x7fdca8060220 2024-01-13T11:56:18.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.662+0000 7fdcadc41700 1 -- 172.21.15.123:0/4283733349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca80f5680 con 0x7fdca8060220 2024-01-13T11:56:18.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.663+0000 7fdc8ffff700 1 -- 172.21.15.123:0/4283733349 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc9801d3d0 con 0x7fdca8060220 2024-01-13T11:56:18.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.663+0000 7fdcadc41700 1 -- 172.21.15.123:0/4283733349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca8059100 con 0x7fdca8060220 2024-01-13T11:56:18.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.663+0000 7fdc8ffff700 1 --2- 172.21.15.123:0/4283733349 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fdc9002ff60 0x7fdc90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:18.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.663+0000 7fdc8ffff700 1 -- 172.21.15.123:0/4283733349 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc980410e0 con 0x7fdca8060220 2024-01-13T11:56:18.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.663+0000 7fdca6ffd700 1 --2- 172.21.15.123:0/4283733349 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fdc9002ff60 0x7fdc90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:18.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.664+0000 7fdca6ffd700 1 --2- 172.21.15.123:0/4283733349 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fdc9002ff60 0x7fdc90032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fdc9c0096a0 tx=0x7fdc9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:18.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.668+0000 7fdc8ffff700 1 -- 172.21.15.123:0/4283733349 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc980126f0 con 0x7fdca8060220 2024-01-13T11:56:18.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.874+0000 7fdcadc41700 1 -- 172.21.15.123:0/4283733349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdca8045ad0 con 0x7fdca8060220 2024-01-13T11:56:18.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.875+0000 7fdc8ffff700 1 -- 172.21.15.123:0/4283733349 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdc98012330 con 0x7fdca8060220 2024-01-13T11:56:18.864 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:18.864 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:18.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.877+0000 7fdcadc41700 1 -- 172.21.15.123:0/4283733349 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fdc9002ff60 msgr2=0x7fdc90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:18.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.877+0000 7fdcadc41700 1 --2- 172.21.15.123:0/4283733349 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fdc9002ff60 0x7fdc90032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fdc9c0096a0 tx=0x7fdc9c006b40).stop 2024-01-13T11:56:18.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.877+0000 7fdcadc41700 1 -- 172.21.15.123:0/4283733349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdca8060220 msgr2=0x7fdca80f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:18.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.877+0000 7fdcadc41700 1 --2- 172.21.15.123:0/4283733349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdca8060220 0x7fdca80f6bd0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fdc98013510 tx=0x7fdc9800b040).stop 2024-01-13T11:56:18.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.877+0000 7fdcadc41700 1 -- 172.21.15.123:0/4283733349 shutdown_connections 2024-01-13T11:56:18.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.877+0000 7fdcadc41700 1 --2- 172.21.15.123:0/4283733349 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fdc9002ff60 0x7fdc90032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:18.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.877+0000 7fdcadc41700 1 --2- 172.21.15.123:0/4283733349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdca8060220 0x7fdca80f6bd0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:18.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.877+0000 7fdcadc41700 1 -- 172.21.15.123:0/4283733349 >> 172.21.15.123:0/4283733349 conn(0x7fdca80f0b70 msgr2=0x7fdca80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:18.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.877+0000 7fdcadc41700 1 -- 172.21.15.123:0/4283733349 shutdown_connections 2024-01-13T11:56:18.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:18.877+0000 7fdcadc41700 1 -- 172.21.15.123:0/4283733349 wait complete. 2024-01-13T11:56:18.866 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:19.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:19 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:17.563466+0000 mgr.smithi059.hmyldj (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:19.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:19 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:18.876590+0000 mon.smithi059 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.123:0/4283733349' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:20.256 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:20.256 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:21.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.383+0000 7feccd53d700 1 -- 172.21.15.123:0/3273710334 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecc80f6510 msgr2=0x7fecc80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:21.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.383+0000 7feccd53d700 1 --2- 172.21.15.123:0/3273710334 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecc80f6510 0x7fecc80f6930 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fecbc009fb0 tx=0x7fecbc009b30).stop 2024-01-13T11:56:21.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.383+0000 7feccd53d700 1 -- 172.21.15.123:0/3273710334 shutdown_connections 2024-01-13T11:56:21.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.383+0000 7feccd53d700 1 --2- 172.21.15.123:0/3273710334 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecc80f6510 0x7fecc80f6930 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:21.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.383+0000 7feccd53d700 1 -- 172.21.15.123:0/3273710334 >> 172.21.15.123:0/3273710334 conn(0x7fecc80f1c70 msgr2=0x7fecc80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:21.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.383+0000 7feccd53d700 1 -- 172.21.15.123:0/3273710334 shutdown_connections 2024-01-13T11:56:21.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.383+0000 7feccd53d700 1 -- 172.21.15.123:0/3273710334 wait complete. 2024-01-13T11:56:21.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.384+0000 7feccd53d700 1 Processor -- start 2024-01-13T11:56:21.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.384+0000 7feccd53d700 1 -- start start 2024-01-13T11:56:21.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.384+0000 7feccd53d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecc80f6510 0x7fecc8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:21.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.384+0000 7feccd53d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc8106c20 con 0x7fecc80f6510 2024-01-13T11:56:21.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.384+0000 7fecc6ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecc80f6510 0x7fecc8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:21.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.384+0000 7fecc6ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecc80f6510 0x7fecc8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:48796/0 (socket says 172.21.15.123:48796) 2024-01-13T11:56:21.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.384+0000 7fecc6ffd700 1 -- 172.21.15.123:0/2398321191 learned_addr learned my addr 172.21.15.123:0/2398321191 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:21.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.384+0000 7fecc6ffd700 1 -- 172.21.15.123:0/2398321191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecbc005040 con 0x7fecc80f6510 2024-01-13T11:56:21.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.384+0000 7fecc6ffd700 1 --2- 172.21.15.123:0/2398321191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecc80f6510 0x7fecc8106730 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fecbc013510 tx=0x7fecbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:21.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.385+0000 7fecaf7fe700 1 -- 172.21.15.123:0/2398321191 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecbc009320 con 0x7fecc80f6510 2024-01-13T11:56:21.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.385+0000 7feccd53d700 1 -- 172.21.15.123:0/2398321191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc8106e20 con 0x7fecc80f6510 2024-01-13T11:56:21.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.385+0000 7fecaf7fe700 1 -- 172.21.15.123:0/2398321191 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fecbc013f00 con 0x7fecc80f6510 2024-01-13T11:56:21.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.385+0000 7fecaf7fe700 1 -- 172.21.15.123:0/2398321191 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecbc014d90 con 0x7fecc80f6510 2024-01-13T11:56:21.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.385+0000 7feccd53d700 1 -- 172.21.15.123:0/2398321191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc81072b0 con 0x7fecc80f6510 2024-01-13T11:56:21.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.385+0000 7fecaf7fe700 1 -- 172.21.15.123:0/2398321191 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fecbc0263a0 con 0x7fecc80f6510 2024-01-13T11:56:21.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.386+0000 7fecaf7fe700 1 --2- 172.21.15.123:0/2398321191 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fecb002ff60 0x7fecb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:21.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.386+0000 7fecaf7fe700 1 -- 172.21.15.123:0/2398321191 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fecbc0410d0 con 0x7fecc80f6510 2024-01-13T11:56:21.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.386+0000 7feccd53d700 1 -- 172.21.15.123:0/2398321191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecc8045ad0 con 0x7fecc80f6510 2024-01-13T11:56:21.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.386+0000 7fecc67fc700 1 --2- 172.21.15.123:0/2398321191 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fecb002ff60 0x7fecb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:21.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.386+0000 7fecc67fc700 1 --2- 172.21.15.123:0/2398321191 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fecb002ff60 0x7fecb0032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fecb80096a0 tx=0x7fecb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:21.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.391+0000 7fecaf7fe700 1 -- 172.21.15.123:0/2398321191 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fecbc012920 con 0x7fecc80f6510 2024-01-13T11:56:21.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.597+0000 7feccd53d700 1 -- 172.21.15.123:0/2398321191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fecc80fbf60 con 0x7fecc80f6510 2024-01-13T11:56:21.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.598+0000 7fecaf7fe700 1 -- 172.21.15.123:0/2398321191 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fecbc02b020 con 0x7fecc80f6510 2024-01-13T11:56:21.586 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:21.586 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:21.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.599+0000 7feccd53d700 1 -- 172.21.15.123:0/2398321191 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fecb002ff60 msgr2=0x7fecb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.600+0000 7feccd53d700 1 --2- 172.21.15.123:0/2398321191 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fecb002ff60 0x7fecb0032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fecb80096a0 tx=0x7fecb8006b40).stop 2024-01-13T11:56:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.600+0000 7feccd53d700 1 -- 172.21.15.123:0/2398321191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecc80f6510 msgr2=0x7fecc8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.600+0000 7feccd53d700 1 --2- 172.21.15.123:0/2398321191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecc80f6510 0x7fecc8106730 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fecbc013510 tx=0x7fecbc00b040).stop 2024-01-13T11:56:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.600+0000 7feccd53d700 1 -- 172.21.15.123:0/2398321191 shutdown_connections 2024-01-13T11:56:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.600+0000 7feccd53d700 1 --2- 172.21.15.123:0/2398321191 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fecb002ff60 0x7fecb0032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.600+0000 7feccd53d700 1 --2- 172.21.15.123:0/2398321191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecc80f6510 0x7fecc8106730 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.600+0000 7feccd53d700 1 -- 172.21.15.123:0/2398321191 >> 172.21.15.123:0/2398321191 conn(0x7fecc80f1c70 msgr2=0x7fecc80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.600+0000 7feccd53d700 1 -- 172.21.15.123:0/2398321191 shutdown_connections 2024-01-13T11:56:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:21.600+0000 7feccd53d700 1 -- 172.21.15.123:0/2398321191 wait complete. 2024-01-13T11:56:21.588 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:21.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:21 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:19.563772+0000 mgr.smithi059.hmyldj (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:22.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:22 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:21.599159+0000 mon.smithi059 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.123:0/2398321191' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:22.955 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:22.956 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:23.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:23 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:21.564117+0000 mgr.smithi059.hmyldj (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:24.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.108+0000 7fe6b50be700 1 -- 172.21.15.123:0/3332605778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6b00f5410 msgr2=0x7fe6b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:24.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.108+0000 7fe6b50be700 1 --2- 172.21.15.123:0/3332605778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6b00f5410 0x7fe6b00f5830 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fe698009fb0 tx=0x7fe698009b30).stop 2024-01-13T11:56:24.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.108+0000 7fe6b50be700 1 -- 172.21.15.123:0/3332605778 shutdown_connections 2024-01-13T11:56:24.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.108+0000 7fe6b50be700 1 --2- 172.21.15.123:0/3332605778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6b00f5410 0x7fe6b00f5830 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:24.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.108+0000 7fe6b50be700 1 -- 172.21.15.123:0/3332605778 >> 172.21.15.123:0/3332605778 conn(0x7fe6b00f0b70 msgr2=0x7fe6b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:24.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.108+0000 7fe6b50be700 1 -- 172.21.15.123:0/3332605778 shutdown_connections 2024-01-13T11:56:24.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.108+0000 7fe6b50be700 1 -- 172.21.15.123:0/3332605778 wait complete. 2024-01-13T11:56:24.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.109+0000 7fe6b50be700 1 Processor -- start 2024-01-13T11:56:24.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.109+0000 7fe6b50be700 1 -- start start 2024-01-13T11:56:24.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.109+0000 7fe6b50be700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6b00f5410 0x7fe6b00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:24.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.109+0000 7fe6b50be700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b00fb820 con 0x7fe6b00f5410 2024-01-13T11:56:24.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.109+0000 7fe6aed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6b00f5410 0x7fe6b00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:24.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.109+0000 7fe6aed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6b00f5410 0x7fe6b00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:48814/0 (socket says 172.21.15.123:48814) 2024-01-13T11:56:24.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.109+0000 7fe6aed9d700 1 -- 172.21.15.123:0/3366085077 learned_addr learned my addr 172.21.15.123:0/3366085077 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:24.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.110+0000 7fe6aed9d700 1 -- 172.21.15.123:0/3366085077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe698005040 con 0x7fe6b00f5410 2024-01-13T11:56:24.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.110+0000 7fe6aed9d700 1 --2- 172.21.15.123:0/3366085077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6b00f5410 0x7fe6b00fd3a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fe698013510 tx=0x7fe69800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:24.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.110+0000 7fe6a77fe700 1 -- 172.21.15.123:0/3366085077 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe698009320 con 0x7fe6b00f5410 2024-01-13T11:56:24.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.110+0000 7fe6b50be700 1 -- 172.21.15.123:0/3366085077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b00fb9c0 con 0x7fe6b00f5410 2024-01-13T11:56:24.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.110+0000 7fe6a77fe700 1 -- 172.21.15.123:0/3366085077 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe698013f00 con 0x7fe6b00f5410 2024-01-13T11:56:24.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.110+0000 7fe6a77fe700 1 -- 172.21.15.123:0/3366085077 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe698014d90 con 0x7fe6b00f5410 2024-01-13T11:56:24.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.110+0000 7fe6b50be700 1 -- 172.21.15.123:0/3366085077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b00fbe50 con 0x7fe6b00f5410 2024-01-13T11:56:24.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.111+0000 7fe6b50be700 1 -- 172.21.15.123:0/3366085077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b0059100 con 0x7fe6b00f5410 2024-01-13T11:56:24.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.111+0000 7fe6a77fe700 1 -- 172.21.15.123:0/3366085077 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe6980095f0 con 0x7fe6b00f5410 2024-01-13T11:56:24.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.111+0000 7fe6a77fe700 1 --2- 172.21.15.123:0/3366085077 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe69c02ffb0 0x7fe69c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:24.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.111+0000 7fe6a77fe700 1 -- 172.21.15.123:0/3366085077 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe698041080 con 0x7fe6b00f5410 2024-01-13T11:56:24.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.111+0000 7fe6ae59c700 1 --2- 172.21.15.123:0/3366085077 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe69c02ffb0 0x7fe69c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:24.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.112+0000 7fe6ae59c700 1 --2- 172.21.15.123:0/3366085077 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe69c02ffb0 0x7fe69c032470 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fe6a00096a0 tx=0x7fe6a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:24.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.116+0000 7fe6a77fe700 1 -- 172.21.15.123:0/3366085077 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe698014420 con 0x7fe6b00f5410 2024-01-13T11:56:24.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.322+0000 7fe6b50be700 1 -- 172.21.15.123:0/3366085077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6b0045ad0 con 0x7fe6b00f5410 2024-01-13T11:56:24.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.323+0000 7fe6a77fe700 1 -- 172.21.15.123:0/3366085077 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe698010230 con 0x7fe6b00f5410 2024-01-13T11:56:24.310 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:24.311 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:24.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.324+0000 7fe6b50be700 1 -- 172.21.15.123:0/3366085077 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe69c02ffb0 msgr2=0x7fe69c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:24.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.324+0000 7fe6b50be700 1 --2- 172.21.15.123:0/3366085077 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe69c02ffb0 0x7fe69c032470 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fe6a00096a0 tx=0x7fe6a0006b40).stop 2024-01-13T11:56:24.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.324+0000 7fe6b50be700 1 -- 172.21.15.123:0/3366085077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6b00f5410 msgr2=0x7fe6b00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:24.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.324+0000 7fe6b50be700 1 --2- 172.21.15.123:0/3366085077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6b00f5410 0x7fe6b00fd3a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fe698013510 tx=0x7fe69800b040).stop 2024-01-13T11:56:24.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.324+0000 7fe6b50be700 1 -- 172.21.15.123:0/3366085077 shutdown_connections 2024-01-13T11:56:24.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.324+0000 7fe6b50be700 1 --2- 172.21.15.123:0/3366085077 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe69c02ffb0 0x7fe69c032470 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:24.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.324+0000 7fe6b50be700 1 --2- 172.21.15.123:0/3366085077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6b00f5410 0x7fe6b00fd3a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:24.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.324+0000 7fe6b50be700 1 -- 172.21.15.123:0/3366085077 >> 172.21.15.123:0/3366085077 conn(0x7fe6b00f0b70 msgr2=0x7fe6b005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:24.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.324+0000 7fe6b50be700 1 -- 172.21.15.123:0/3366085077 shutdown_connections 2024-01-13T11:56:24.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:24.324+0000 7fe6b50be700 1 -- 172.21.15.123:0/3366085077 wait complete. 2024-01-13T11:56:24.313 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:25.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:25 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:23.564587+0000 mgr.smithi059.hmyldj (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:25.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:25 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:24.323926+0000 mon.smithi059 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.123:0/3366085077' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:25.678 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:25.678 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:26.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.829+0000 7fabc2b1f700 1 -- 172.21.15.123:0/2580352748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabbc0f62a0 msgr2=0x7fabbc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:26.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.829+0000 7fabc2b1f700 1 --2- 172.21.15.123:0/2580352748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabbc0f62a0 0x7fabbc0f66c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fabb8009fb0 tx=0x7fabb8009b30).stop 2024-01-13T11:56:26.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.830+0000 7fabc2b1f700 1 -- 172.21.15.123:0/2580352748 shutdown_connections 2024-01-13T11:56:26.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.830+0000 7fabc2b1f700 1 --2- 172.21.15.123:0/2580352748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabbc0f62a0 0x7fabbc0f66c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:26.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.830+0000 7fabc2b1f700 1 -- 172.21.15.123:0/2580352748 >> 172.21.15.123:0/2580352748 conn(0x7fabbc0f1aa0 msgr2=0x7fabbc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:26.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.830+0000 7fabc2b1f700 1 -- 172.21.15.123:0/2580352748 shutdown_connections 2024-01-13T11:56:26.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.830+0000 7fabc2b1f700 1 -- 172.21.15.123:0/2580352748 wait complete. 2024-01-13T11:56:26.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.830+0000 7fabc2b1f700 1 Processor -- start 2024-01-13T11:56:26.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.830+0000 7fabc2b1f700 1 -- start start 2024-01-13T11:56:26.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.831+0000 7fabc2b1f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabbc0f62a0 0x7fabbc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:26.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.831+0000 7fabc2b1f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabbc1069a0 con 0x7fabbc0f62a0 2024-01-13T11:56:26.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.831+0000 7fabc1b1d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabbc0f62a0 0x7fabbc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:26.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.831+0000 7fabc1b1d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabbc0f62a0 0x7fabbc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:60338/0 (socket says 172.21.15.123:60338) 2024-01-13T11:56:26.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.831+0000 7fabc1b1d700 1 -- 172.21.15.123:0/2090202009 learned_addr learned my addr 172.21.15.123:0/2090202009 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:26.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.831+0000 7fabc1b1d700 1 -- 172.21.15.123:0/2090202009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb8005040 con 0x7fabbc0f62a0 2024-01-13T11:56:26.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.831+0000 7fabc1b1d700 1 --2- 172.21.15.123:0/2090202009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabbc0f62a0 0x7fabbc1064b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fabb8013510 tx=0x7fabb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:26.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.831+0000 7fabb27fc700 1 -- 172.21.15.123:0/2090202009 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabb8009320 con 0x7fabbc0f62a0 2024-01-13T11:56:26.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.831+0000 7fabc2b1f700 1 -- 172.21.15.123:0/2090202009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabbc106ba0 con 0x7fabbc0f62a0 2024-01-13T11:56:26.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.831+0000 7fabb27fc700 1 -- 172.21.15.123:0/2090202009 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fabb8013f00 con 0x7fabbc0f62a0 2024-01-13T11:56:26.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.831+0000 7fabb27fc700 1 -- 172.21.15.123:0/2090202009 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabb8014dc0 con 0x7fabbc0f62a0 2024-01-13T11:56:26.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.831+0000 7fabc2b1f700 1 -- 172.21.15.123:0/2090202009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabbc107030 con 0x7fabbc0f62a0 2024-01-13T11:56:26.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.832+0000 7fabb27fc700 1 -- 172.21.15.123:0/2090202009 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fabb80095f0 con 0x7fabbc0f62a0 2024-01-13T11:56:26.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.832+0000 7fabc2b1f700 1 -- 172.21.15.123:0/2090202009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabbc045ad0 con 0x7fabbc0f62a0 2024-01-13T11:56:26.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.832+0000 7fabb27fc700 1 --2- 172.21.15.123:0/2090202009 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7faba802ff60 0x7faba8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:26.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.833+0000 7fabb27fc700 1 -- 172.21.15.123:0/2090202009 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fabb8041000 con 0x7fabbc0f62a0 2024-01-13T11:56:26.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.833+0000 7fabc131c700 1 --2- 172.21.15.123:0/2090202009 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7faba802ff60 0x7faba8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:26.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.833+0000 7fabc131c700 1 --2- 172.21.15.123:0/2090202009 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7faba802ff60 0x7faba8032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fabac0096a0 tx=0x7fabac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:26.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:26.837+0000 7fabb27fc700 1 -- 172.21.15.123:0/2090202009 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fabb801c790 con 0x7fabbc0f62a0 2024-01-13T11:56:27.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:27.042+0000 7fabc2b1f700 1 -- 172.21.15.123:0/2090202009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fabbc107ee0 con 0x7fabbc0f62a0 2024-01-13T11:56:27.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:27.043+0000 7fabb27fc700 1 -- 172.21.15.123:0/2090202009 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fabb8010230 con 0x7fabbc0f62a0 2024-01-13T11:56:27.031 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:27.031 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:27.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:27.045+0000 7fabc2b1f700 1 -- 172.21.15.123:0/2090202009 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7faba802ff60 msgr2=0x7faba8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:27.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:27.045+0000 7fabc2b1f700 1 --2- 172.21.15.123:0/2090202009 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7faba802ff60 0x7faba8032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fabac0096a0 tx=0x7fabac006b40).stop 2024-01-13T11:56:27.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:27.045+0000 7fabc2b1f700 1 -- 172.21.15.123:0/2090202009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabbc0f62a0 msgr2=0x7fabbc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:27.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:27.045+0000 7fabc2b1f700 1 --2- 172.21.15.123:0/2090202009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabbc0f62a0 0x7fabbc1064b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fabb8013510 tx=0x7fabb800b040).stop 2024-01-13T11:56:27.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:27.045+0000 7fabc2b1f700 1 -- 172.21.15.123:0/2090202009 shutdown_connections 2024-01-13T11:56:27.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:27.045+0000 7fabc2b1f700 1 --2- 172.21.15.123:0/2090202009 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7faba802ff60 0x7faba8032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:27.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:27.045+0000 7fabc2b1f700 1 --2- 172.21.15.123:0/2090202009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabbc0f62a0 0x7fabbc1064b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:27.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:27.045+0000 7fabc2b1f700 1 -- 172.21.15.123:0/2090202009 >> 172.21.15.123:0/2090202009 conn(0x7fabbc0f1aa0 msgr2=0x7fabbc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:27.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:27.045+0000 7fabc2b1f700 1 -- 172.21.15.123:0/2090202009 shutdown_connections 2024-01-13T11:56:27.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:27.045+0000 7fabc2b1f700 1 -- 172.21.15.123:0/2090202009 wait complete. 2024-01-13T11:56:27.034 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:27.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:27 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:25.564966+0000 mgr.smithi059.hmyldj (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:27.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:27 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:27.044390+0000 mon.smithi059 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.123:0/2090202009' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:28.405 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:28.405 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:29.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.604+0000 7f2a6c05e700 1 -- 172.21.15.123:0/4047224000 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a640f6480 msgr2=0x7f2a640f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:29.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.604+0000 7f2a6c05e700 1 --2- 172.21.15.123:0/4047224000 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a640f6480 0x7f2a640f68a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f2a54009fb0 tx=0x7f2a54009b30).stop 2024-01-13T11:56:29.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.604+0000 7f2a6c05e700 1 -- 172.21.15.123:0/4047224000 shutdown_connections 2024-01-13T11:56:29.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.604+0000 7f2a6c05e700 1 --2- 172.21.15.123:0/4047224000 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a640f6480 0x7f2a640f68a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:29.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.604+0000 7f2a6c05e700 1 -- 172.21.15.123:0/4047224000 >> 172.21.15.123:0/4047224000 conn(0x7f2a640f1c00 msgr2=0x7f2a640f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:29.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.604+0000 7f2a6c05e700 1 -- 172.21.15.123:0/4047224000 shutdown_connections 2024-01-13T11:56:29.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.604+0000 7f2a6c05e700 1 -- 172.21.15.123:0/4047224000 wait complete. 2024-01-13T11:56:29.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.605+0000 7f2a6c05e700 1 Processor -- start 2024-01-13T11:56:29.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.605+0000 7f2a6c05e700 1 -- start start 2024-01-13T11:56:29.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.605+0000 7f2a6c05e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a64106810 0x7f2a64106c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:29.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.605+0000 7f2a6c05e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a64107120 con 0x7f2a64106810 2024-01-13T11:56:29.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.605+0000 7f2a69dfa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a64106810 0x7f2a64106c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:29.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.605+0000 7f2a69dfa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a64106810 0x7f2a64106c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:60350/0 (socket says 172.21.15.123:60350) 2024-01-13T11:56:29.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.605+0000 7f2a69dfa700 1 -- 172.21.15.123:0/533982967 learned_addr learned my addr 172.21.15.123:0/533982967 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:29.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.606+0000 7f2a69dfa700 1 -- 172.21.15.123:0/533982967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a54005040 con 0x7f2a64106810 2024-01-13T11:56:29.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.606+0000 7f2a69dfa700 1 --2- 172.21.15.123:0/533982967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a64106810 0x7f2a64106c30 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f2a54009f80 tx=0x7f2a54009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:29.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.606+0000 7f2a5a7fc700 1 -- 172.21.15.123:0/533982967 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a540136c0 con 0x7f2a64106810 2024-01-13T11:56:29.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.606+0000 7f2a6c05e700 1 -- 172.21.15.123:0/533982967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a64107320 con 0x7f2a64106810 2024-01-13T11:56:29.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.606+0000 7f2a5a7fc700 1 -- 172.21.15.123:0/533982967 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a540095f0 con 0x7f2a64106810 2024-01-13T11:56:29.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.606+0000 7f2a5a7fc700 1 -- 172.21.15.123:0/533982967 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a54014420 con 0x7f2a64106810 2024-01-13T11:56:29.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.606+0000 7f2a6c05e700 1 -- 172.21.15.123:0/533982967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a64107e70 con 0x7f2a64106810 2024-01-13T11:56:29.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.607+0000 7f2a5a7fc700 1 -- 172.21.15.123:0/533982967 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2a54025060 con 0x7f2a64106810 2024-01-13T11:56:29.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.607+0000 7f2a6c05e700 1 -- 172.21.15.123:0/533982967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a64059100 con 0x7f2a64106810 2024-01-13T11:56:29.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.607+0000 7f2a5a7fc700 1 --2- 172.21.15.123:0/533982967 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f2a5002ff60 0x7f2a50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:29.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.607+0000 7f2a5a7fc700 1 -- 172.21.15.123:0/533982967 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2a54040e60 con 0x7f2a64106810 2024-01-13T11:56:29.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.607+0000 7f2a695f9700 1 --2- 172.21.15.123:0/533982967 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f2a5002ff60 0x7f2a50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:29.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.608+0000 7f2a695f9700 1 --2- 172.21.15.123:0/533982967 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f2a5002ff60 0x7f2a50032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f2a600096a0 tx=0x7f2a60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:29.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.612+0000 7f2a5a7fc700 1 -- 172.21.15.123:0/533982967 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a5401cb60 con 0x7f2a64106810 2024-01-13T11:56:29.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:29 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:27.565301+0000 mgr.smithi059.hmyldj (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:29.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.816+0000 7f2a6c05e700 1 -- 172.21.15.123:0/533982967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a64045ad0 con 0x7f2a64106810 2024-01-13T11:56:29.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.817+0000 7f2a5a7fc700 1 -- 172.21.15.123:0/533982967 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2a5400f280 con 0x7f2a64106810 2024-01-13T11:56:29.805 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:29.805 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:29.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.818+0000 7f2a6c05e700 1 -- 172.21.15.123:0/533982967 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f2a5002ff60 msgr2=0x7f2a50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:29.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.818+0000 7f2a6c05e700 1 --2- 172.21.15.123:0/533982967 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f2a5002ff60 0x7f2a50032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f2a600096a0 tx=0x7f2a60006b40).stop 2024-01-13T11:56:29.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.818+0000 7f2a6c05e700 1 -- 172.21.15.123:0/533982967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a64106810 msgr2=0x7f2a64106c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:29.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.818+0000 7f2a6c05e700 1 --2- 172.21.15.123:0/533982967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a64106810 0x7f2a64106c30 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f2a54009f80 tx=0x7f2a54009b30).stop 2024-01-13T11:56:29.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.818+0000 7f2a6c05e700 1 -- 172.21.15.123:0/533982967 shutdown_connections 2024-01-13T11:56:29.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.818+0000 7f2a6c05e700 1 --2- 172.21.15.123:0/533982967 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f2a5002ff60 0x7f2a50032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:29.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.818+0000 7f2a6c05e700 1 --2- 172.21.15.123:0/533982967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a64106810 0x7f2a64106c30 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:29.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.818+0000 7f2a6c05e700 1 -- 172.21.15.123:0/533982967 >> 172.21.15.123:0/533982967 conn(0x7f2a640f1c00 msgr2=0x7f2a640f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:29.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.818+0000 7f2a6c05e700 1 -- 172.21.15.123:0/533982967 shutdown_connections 2024-01-13T11:56:29.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:29.818+0000 7f2a6c05e700 1 -- 172.21.15.123:0/533982967 wait complete. 2024-01-13T11:56:29.807 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:30.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:30 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:29.818365+0000 mon.smithi059 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.123:0/533982967' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:31.187 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:31.188 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:31.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:31 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:29.565716+0000 mgr.smithi059.hmyldj (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:32.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.317+0000 7fb0322ce700 1 -- 172.21.15.123:0/2554661811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb02c0f5410 msgr2=0x7fb02c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:32.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.317+0000 7fb0322ce700 1 --2- 172.21.15.123:0/2554661811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb02c0f5410 0x7fb02c0f5830 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fb014009fb0 tx=0x7fb014009b30).stop 2024-01-13T11:56:32.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.317+0000 7fb0322ce700 1 -- 172.21.15.123:0/2554661811 shutdown_connections 2024-01-13T11:56:32.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.317+0000 7fb0322ce700 1 --2- 172.21.15.123:0/2554661811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb02c0f5410 0x7fb02c0f5830 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:32.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.317+0000 7fb0322ce700 1 -- 172.21.15.123:0/2554661811 >> 172.21.15.123:0/2554661811 conn(0x7fb02c0f0b70 msgr2=0x7fb02c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:32.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.317+0000 7fb0322ce700 1 -- 172.21.15.123:0/2554661811 shutdown_connections 2024-01-13T11:56:32.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.317+0000 7fb0322ce700 1 -- 172.21.15.123:0/2554661811 wait complete. 2024-01-13T11:56:32.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.318+0000 7fb0322ce700 1 Processor -- start 2024-01-13T11:56:32.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.318+0000 7fb0322ce700 1 -- start start 2024-01-13T11:56:32.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.318+0000 7fb0322ce700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb02c0f5410 0x7fb02c106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:32.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.318+0000 7fb0322ce700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c106c00 con 0x7fb02c0f5410 2024-01-13T11:56:32.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.318+0000 7fb02bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb02c0f5410 0x7fb02c106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:32.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.318+0000 7fb02bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb02c0f5410 0x7fb02c106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:60372/0 (socket says 172.21.15.123:60372) 2024-01-13T11:56:32.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.318+0000 7fb02bfff700 1 -- 172.21.15.123:0/3292492120 learned_addr learned my addr 172.21.15.123:0/3292492120 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:32.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.318+0000 7fb02bfff700 1 -- 172.21.15.123:0/3292492120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb014005040 con 0x7fb02c0f5410 2024-01-13T11:56:32.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.319+0000 7fb02bfff700 1 --2- 172.21.15.123:0/3292492120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb02c0f5410 0x7fb02c106710 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fb014013510 tx=0x7fb01400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:32.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.319+0000 7fb028ff9700 1 -- 172.21.15.123:0/3292492120 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb014009320 con 0x7fb02c0f5410 2024-01-13T11:56:32.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.319+0000 7fb0322ce700 1 -- 172.21.15.123:0/3292492120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb02c106e00 con 0x7fb02c0f5410 2024-01-13T11:56:32.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.319+0000 7fb028ff9700 1 -- 172.21.15.123:0/3292492120 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb014013f00 con 0x7fb02c0f5410 2024-01-13T11:56:32.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.319+0000 7fb028ff9700 1 -- 172.21.15.123:0/3292492120 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb014014d90 con 0x7fb02c0f5410 2024-01-13T11:56:32.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.319+0000 7fb0322ce700 1 -- 172.21.15.123:0/3292492120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb02c107290 con 0x7fb02c0f5410 2024-01-13T11:56:32.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.320+0000 7fb0322ce700 1 -- 172.21.15.123:0/3292492120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02c045ad0 con 0x7fb02c0f5410 2024-01-13T11:56:32.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.320+0000 7fb028ff9700 1 -- 172.21.15.123:0/3292492120 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0140095f0 con 0x7fb02c0f5410 2024-01-13T11:56:32.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.320+0000 7fb028ff9700 1 --2- 172.21.15.123:0/3292492120 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb01802ff60 0x7fb018032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:32.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.320+0000 7fb028ff9700 1 -- 172.21.15.123:0/3292492120 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb0140410a0 con 0x7fb02c0f5410 2024-01-13T11:56:32.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.320+0000 7fb02b7fe700 1 --2- 172.21.15.123:0/3292492120 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb01802ff60 0x7fb018032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:32.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.320+0000 7fb02b7fe700 1 --2- 172.21.15.123:0/3292492120 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb01802ff60 0x7fb018032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fb01c0096a0 tx=0x7fb01c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:32.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.324+0000 7fb028ff9700 1 -- 172.21.15.123:0/3292492120 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb014010230 con 0x7fb02c0f5410 2024-01-13T11:56:32.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.531+0000 7fb0322ce700 1 -- 172.21.15.123:0/3292492120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb02c1080e0 con 0x7fb02c0f5410 2024-01-13T11:56:32.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.532+0000 7fb028ff9700 1 -- 172.21.15.123:0/3292492120 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb014010230 con 0x7fb02c0f5410 2024-01-13T11:56:32.520 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:32.520 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:32.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.533+0000 7fb0322ce700 1 -- 172.21.15.123:0/3292492120 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb01802ff60 msgr2=0x7fb018032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:32.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.533+0000 7fb0322ce700 1 --2- 172.21.15.123:0/3292492120 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb01802ff60 0x7fb018032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fb01c0096a0 tx=0x7fb01c006b40).stop 2024-01-13T11:56:32.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.534+0000 7fb0322ce700 1 -- 172.21.15.123:0/3292492120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb02c0f5410 msgr2=0x7fb02c106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:32.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.534+0000 7fb0322ce700 1 --2- 172.21.15.123:0/3292492120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb02c0f5410 0x7fb02c106710 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fb014013510 tx=0x7fb01400b040).stop 2024-01-13T11:56:32.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.534+0000 7fb0322ce700 1 -- 172.21.15.123:0/3292492120 shutdown_connections 2024-01-13T11:56:32.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.534+0000 7fb0322ce700 1 --2- 172.21.15.123:0/3292492120 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb01802ff60 0x7fb018032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:32.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.534+0000 7fb0322ce700 1 --2- 172.21.15.123:0/3292492120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb02c0f5410 0x7fb02c106710 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:32.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.534+0000 7fb0322ce700 1 -- 172.21.15.123:0/3292492120 >> 172.21.15.123:0/3292492120 conn(0x7fb02c0f0b70 msgr2=0x7fb02c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:32.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.534+0000 7fb0322ce700 1 -- 172.21.15.123:0/3292492120 shutdown_connections 2024-01-13T11:56:32.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:32.534+0000 7fb0322ce700 1 -- 172.21.15.123:0/3292492120 wait complete. 2024-01-13T11:56:32.522 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:33.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:33 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:31.566102+0000 mgr.smithi059.hmyldj (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:33.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:33 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:32.533282+0000 mon.smithi059 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.123:0/3292492120' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:33.943 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:33.944 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:35.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.146+0000 7fa9e7292700 1 -- 172.21.15.123:0/1254084924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e00f5410 msgr2=0x7fa9e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:35.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.146+0000 7fa9e7292700 1 --2- 172.21.15.123:0/1254084924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e00f5410 0x7fa9e00f5830 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fa9d0009fb0 tx=0x7fa9d0009b30).stop 2024-01-13T11:56:35.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.146+0000 7fa9e7292700 1 -- 172.21.15.123:0/1254084924 shutdown_connections 2024-01-13T11:56:35.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.146+0000 7fa9e7292700 1 --2- 172.21.15.123:0/1254084924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e00f5410 0x7fa9e00f5830 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:35.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.146+0000 7fa9e7292700 1 -- 172.21.15.123:0/1254084924 >> 172.21.15.123:0/1254084924 conn(0x7fa9e00f0b70 msgr2=0x7fa9e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:35.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.146+0000 7fa9e7292700 1 -- 172.21.15.123:0/1254084924 shutdown_connections 2024-01-13T11:56:35.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.146+0000 7fa9e7292700 1 -- 172.21.15.123:0/1254084924 wait complete. 2024-01-13T11:56:35.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.147+0000 7fa9e7292700 1 Processor -- start 2024-01-13T11:56:35.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.147+0000 7fa9e7292700 1 -- start start 2024-01-13T11:56:35.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.147+0000 7fa9e7292700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e00f5410 0x7fa9e00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:35.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.147+0000 7fa9e7292700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e00fb820 con 0x7fa9e00f5410 2024-01-13T11:56:35.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.147+0000 7fa9e502e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e00f5410 0x7fa9e00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:35.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.147+0000 7fa9e502e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e00f5410 0x7fa9e00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:60402/0 (socket says 172.21.15.123:60402) 2024-01-13T11:56:35.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.147+0000 7fa9e502e700 1 -- 172.21.15.123:0/2750770104 learned_addr learned my addr 172.21.15.123:0/2750770104 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:35.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.147+0000 7fa9e502e700 1 -- 172.21.15.123:0/2750770104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d0005040 con 0x7fa9e00f5410 2024-01-13T11:56:35.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.148+0000 7fa9e502e700 1 --2- 172.21.15.123:0/2750770104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e00f5410 0x7fa9e00fd3a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fa9d0013510 tx=0x7fa9d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:35.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.148+0000 7fa9d5ffb700 1 -- 172.21.15.123:0/2750770104 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9d0009320 con 0x7fa9e00f5410 2024-01-13T11:56:35.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.148+0000 7fa9e7292700 1 -- 172.21.15.123:0/2750770104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e00fb9c0 con 0x7fa9e00f5410 2024-01-13T11:56:35.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.148+0000 7fa9d5ffb700 1 -- 172.21.15.123:0/2750770104 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa9d0013f00 con 0x7fa9e00f5410 2024-01-13T11:56:35.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.148+0000 7fa9d5ffb700 1 -- 172.21.15.123:0/2750770104 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9d0014d90 con 0x7fa9e00f5410 2024-01-13T11:56:35.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.148+0000 7fa9e7292700 1 -- 172.21.15.123:0/2750770104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e00fbe50 con 0x7fa9e00f5410 2024-01-13T11:56:35.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.149+0000 7fa9d5ffb700 1 -- 172.21.15.123:0/2750770104 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9d00095f0 con 0x7fa9e00f5410 2024-01-13T11:56:35.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.149+0000 7fa9e7292700 1 -- 172.21.15.123:0/2750770104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e0059100 con 0x7fa9e00f5410 2024-01-13T11:56:35.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.149+0000 7fa9d5ffb700 1 --2- 172.21.15.123:0/2750770104 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa9cc02ffb0 0x7fa9cc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:35.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.149+0000 7fa9d5ffb700 1 -- 172.21.15.123:0/2750770104 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa9d0040f80 con 0x7fa9e00f5410 2024-01-13T11:56:35.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.149+0000 7fa9e482d700 1 --2- 172.21.15.123:0/2750770104 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa9cc02ffb0 0x7fa9cc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:35.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.149+0000 7fa9e482d700 1 --2- 172.21.15.123:0/2750770104 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa9cc02ffb0 0x7fa9cc032470 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fa9dc0096a0 tx=0x7fa9dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:35.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.154+0000 7fa9d5ffb700 1 -- 172.21.15.123:0/2750770104 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9d0014420 con 0x7fa9e00f5410 2024-01-13T11:56:35.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.359+0000 7fa9e7292700 1 -- 172.21.15.123:0/2750770104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9e00fae60 con 0x7fa9e00f5410 2024-01-13T11:56:35.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.361+0000 7fa9d5ffb700 1 -- 172.21.15.123:0/2750770104 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa9d001d7f0 con 0x7fa9e00f5410 2024-01-13T11:56:35.348 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:35.348 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:35.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.361+0000 7fa9e7292700 1 -- 172.21.15.123:0/2750770104 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa9cc02ffb0 msgr2=0x7fa9cc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:35.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.362+0000 7fa9e7292700 1 --2- 172.21.15.123:0/2750770104 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa9cc02ffb0 0x7fa9cc032470 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fa9dc0096a0 tx=0x7fa9dc006b40).stop 2024-01-13T11:56:35.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.362+0000 7fa9e7292700 1 -- 172.21.15.123:0/2750770104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e00f5410 msgr2=0x7fa9e00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:35.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.362+0000 7fa9e7292700 1 --2- 172.21.15.123:0/2750770104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e00f5410 0x7fa9e00fd3a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fa9d0013510 tx=0x7fa9d000b040).stop 2024-01-13T11:56:35.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.362+0000 7fa9e7292700 1 -- 172.21.15.123:0/2750770104 shutdown_connections 2024-01-13T11:56:35.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.362+0000 7fa9e7292700 1 --2- 172.21.15.123:0/2750770104 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa9cc02ffb0 0x7fa9cc032470 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:35.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.362+0000 7fa9e7292700 1 --2- 172.21.15.123:0/2750770104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9e00f5410 0x7fa9e00fd3a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:35.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.362+0000 7fa9e7292700 1 -- 172.21.15.123:0/2750770104 >> 172.21.15.123:0/2750770104 conn(0x7fa9e00f0b70 msgr2=0x7fa9e005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:35.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.362+0000 7fa9e7292700 1 -- 172.21.15.123:0/2750770104 shutdown_connections 2024-01-13T11:56:35.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:35.362+0000 7fa9e7292700 1 -- 172.21.15.123:0/2750770104 wait complete. 2024-01-13T11:56:35.350 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:35.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:35 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:33.566519+0000 mgr.smithi059.hmyldj (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:36.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:36 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:35.361356+0000 mon.smithi059 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.123:0/2750770104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:36.741 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:36.741 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:37.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:37 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:35.566787+0000 mgr.smithi059.hmyldj (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:37.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.971+0000 7fce8af09700 1 -- 172.21.15.123:0/3309073198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce840f6510 msgr2=0x7fce840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:37.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.971+0000 7fce8af09700 1 --2- 172.21.15.123:0/3309073198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce840f6510 0x7fce840f6930 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fce78009fb0 tx=0x7fce78009b30).stop 2024-01-13T11:56:37.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.971+0000 7fce8af09700 1 -- 172.21.15.123:0/3309073198 shutdown_connections 2024-01-13T11:56:37.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.971+0000 7fce8af09700 1 --2- 172.21.15.123:0/3309073198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce840f6510 0x7fce840f6930 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:37.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.971+0000 7fce8af09700 1 -- 172.21.15.123:0/3309073198 >> 172.21.15.123:0/3309073198 conn(0x7fce840f1c70 msgr2=0x7fce840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:37.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.971+0000 7fce8af09700 1 -- 172.21.15.123:0/3309073198 shutdown_connections 2024-01-13T11:56:37.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.971+0000 7fce8af09700 1 -- 172.21.15.123:0/3309073198 wait complete. 2024-01-13T11:56:37.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.972+0000 7fce8af09700 1 Processor -- start 2024-01-13T11:56:37.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.972+0000 7fce8af09700 1 -- start start 2024-01-13T11:56:37.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.972+0000 7fce8af09700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce840f6510 0x7fce84106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:37.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.972+0000 7fce8af09700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce84106c20 con 0x7fce840f6510 2024-01-13T11:56:37.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.972+0000 7fce88ca5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce840f6510 0x7fce84106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:37.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.972+0000 7fce88ca5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce840f6510 0x7fce84106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:34780/0 (socket says 172.21.15.123:34780) 2024-01-13T11:56:37.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.972+0000 7fce88ca5700 1 -- 172.21.15.123:0/1401236795 learned_addr learned my addr 172.21.15.123:0/1401236795 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:37.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.973+0000 7fce88ca5700 1 -- 172.21.15.123:0/1401236795 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce78005040 con 0x7fce840f6510 2024-01-13T11:56:37.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.973+0000 7fce88ca5700 1 --2- 172.21.15.123:0/1401236795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce840f6510 0x7fce84106730 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fce78013510 tx=0x7fce7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:37.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.973+0000 7fce817fa700 1 -- 172.21.15.123:0/1401236795 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce78009320 con 0x7fce840f6510 2024-01-13T11:56:37.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.973+0000 7fce8af09700 1 -- 172.21.15.123:0/1401236795 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce84106e20 con 0x7fce840f6510 2024-01-13T11:56:37.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.973+0000 7fce817fa700 1 -- 172.21.15.123:0/1401236795 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce78013f00 con 0x7fce840f6510 2024-01-13T11:56:37.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.973+0000 7fce817fa700 1 -- 172.21.15.123:0/1401236795 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce78014d90 con 0x7fce840f6510 2024-01-13T11:56:37.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.973+0000 7fce8af09700 1 -- 172.21.15.123:0/1401236795 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce841072b0 con 0x7fce840f6510 2024-01-13T11:56:37.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.974+0000 7fce817fa700 1 -- 172.21.15.123:0/1401236795 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce780095f0 con 0x7fce840f6510 2024-01-13T11:56:37.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.974+0000 7fce8af09700 1 -- 172.21.15.123:0/1401236795 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce84045ad0 con 0x7fce840f6510 2024-01-13T11:56:37.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.974+0000 7fce817fa700 1 --2- 172.21.15.123:0/1401236795 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce7402ff60 0x7fce74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:37.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.974+0000 7fce817fa700 1 -- 172.21.15.123:0/1401236795 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce78041000 con 0x7fce840f6510 2024-01-13T11:56:37.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.975+0000 7fce83fff700 1 --2- 172.21.15.123:0/1401236795 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce7402ff60 0x7fce74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:37.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.975+0000 7fce83fff700 1 --2- 172.21.15.123:0/1401236795 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce7402ff60 0x7fce74032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fce700096a0 tx=0x7fce70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:37.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:37.979+0000 7fce817fa700 1 -- 172.21.15.123:0/1401236795 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce78014420 con 0x7fce840f6510 2024-01-13T11:56:38.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:38.183+0000 7fce8af09700 1 -- 172.21.15.123:0/1401236795 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce840fbf60 con 0x7fce840f6510 2024-01-13T11:56:38.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:38.183+0000 7fce817fa700 1 -- 172.21.15.123:0/1401236795 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce7801d7f0 con 0x7fce840f6510 2024-01-13T11:56:38.171 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:38.172 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:38.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:38.185+0000 7fce8af09700 1 -- 172.21.15.123:0/1401236795 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce7402ff60 msgr2=0x7fce74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:38.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:38.185+0000 7fce8af09700 1 --2- 172.21.15.123:0/1401236795 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce7402ff60 0x7fce74032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fce700096a0 tx=0x7fce70006b40).stop 2024-01-13T11:56:38.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:38.185+0000 7fce8af09700 1 -- 172.21.15.123:0/1401236795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce840f6510 msgr2=0x7fce84106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:38.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:38.185+0000 7fce8af09700 1 --2- 172.21.15.123:0/1401236795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce840f6510 0x7fce84106730 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fce78013510 tx=0x7fce7800b040).stop 2024-01-13T11:56:38.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:38.185+0000 7fce8af09700 1 -- 172.21.15.123:0/1401236795 shutdown_connections 2024-01-13T11:56:38.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:38.185+0000 7fce8af09700 1 --2- 172.21.15.123:0/1401236795 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce7402ff60 0x7fce74032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:38.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:38.185+0000 7fce8af09700 1 --2- 172.21.15.123:0/1401236795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce840f6510 0x7fce84106730 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:38.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:38.185+0000 7fce8af09700 1 -- 172.21.15.123:0/1401236795 >> 172.21.15.123:0/1401236795 conn(0x7fce840f1c70 msgr2=0x7fce840f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:38.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:38.185+0000 7fce8af09700 1 -- 172.21.15.123:0/1401236795 shutdown_connections 2024-01-13T11:56:38.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:38.185+0000 7fce8af09700 1 -- 172.21.15.123:0/1401236795 wait complete. 2024-01-13T11:56:38.174 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:38.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:38 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:38.184727+0000 mon.smithi059 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.123:0/1401236795' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:39.551 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:39.551 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:39.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:39 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:37.567118+0000 mgr.smithi059.hmyldj (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:40.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:39.644395+0000 mon.smithi059 (mon.0) 277 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/mirror_snapshot_schedule"}]: dispatch 2024-01-13T11:56:40.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:39.644755+0000 mon.smithi059 (mon.0) 278 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/trash_purge_schedule"}]: dispatch 2024-01-13T11:56:40.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.712+0000 7f3ec70ac700 1 -- 172.21.15.123:0/4205161841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ec0060280 msgr2=0x7f3ec00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:40.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.712+0000 7f3ec70ac700 1 --2- 172.21.15.123:0/4205161841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ec0060280 0x7f3ec00f90f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f3eb4009fb0 tx=0x7f3eb4009b30).stop 2024-01-13T11:56:40.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.712+0000 7f3ec70ac700 1 -- 172.21.15.123:0/4205161841 shutdown_connections 2024-01-13T11:56:40.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.712+0000 7f3ec70ac700 1 --2- 172.21.15.123:0/4205161841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ec0060280 0x7f3ec00f90f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:40.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.712+0000 7f3ec70ac700 1 -- 172.21.15.123:0/4205161841 >> 172.21.15.123:0/4205161841 conn(0x7f3ec00f0b70 msgr2=0x7f3ec00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:40.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.712+0000 7f3ec70ac700 1 -- 172.21.15.123:0/4205161841 shutdown_connections 2024-01-13T11:56:40.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.712+0000 7f3ec70ac700 1 -- 172.21.15.123:0/4205161841 wait complete. 2024-01-13T11:56:40.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.713+0000 7f3ec70ac700 1 Processor -- start 2024-01-13T11:56:40.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.713+0000 7f3ec70ac700 1 -- start start 2024-01-13T11:56:40.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.713+0000 7f3ec70ac700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ec0060280 0x7f3ec01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:40.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.713+0000 7f3ec70ac700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec0106bd0 con 0x7f3ec0060280 2024-01-13T11:56:40.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.714+0000 7f3ec4e48700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ec0060280 0x7f3ec01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:40.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.714+0000 7f3ec4e48700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ec0060280 0x7f3ec01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:34798/0 (socket says 172.21.15.123:34798) 2024-01-13T11:56:40.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.714+0000 7f3ec4e48700 1 -- 172.21.15.123:0/1420300442 learned_addr learned my addr 172.21.15.123:0/1420300442 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:40.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.714+0000 7f3ec4e48700 1 -- 172.21.15.123:0/1420300442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb4005040 con 0x7f3ec0060280 2024-01-13T11:56:40.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.714+0000 7f3ec4e48700 1 --2- 172.21.15.123:0/1420300442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ec0060280 0x7f3ec01066e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f3eb4013510 tx=0x7f3eb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:40.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.714+0000 7f3ebd7fa700 1 -- 172.21.15.123:0/1420300442 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3eb4009320 con 0x7f3ec0060280 2024-01-13T11:56:40.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.714+0000 7f3ec70ac700 1 -- 172.21.15.123:0/1420300442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec0106dd0 con 0x7f3ec0060280 2024-01-13T11:56:40.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.714+0000 7f3ebd7fa700 1 -- 172.21.15.123:0/1420300442 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3eb4013f00 con 0x7f3ec0060280 2024-01-13T11:56:40.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.714+0000 7f3ebd7fa700 1 -- 172.21.15.123:0/1420300442 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3eb4014dc0 con 0x7f3ec0060280 2024-01-13T11:56:40.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.714+0000 7f3ec70ac700 1 -- 172.21.15.123:0/1420300442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec0107260 con 0x7f3ec0060280 2024-01-13T11:56:40.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.715+0000 7f3ebd7fa700 1 -- 172.21.15.123:0/1420300442 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3eb40095f0 con 0x7f3ec0060280 2024-01-13T11:56:40.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.715+0000 7f3ec70ac700 1 -- 172.21.15.123:0/1420300442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec0045ad0 con 0x7f3ec0060280 2024-01-13T11:56:40.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.715+0000 7f3ebd7fa700 1 --2- 172.21.15.123:0/1420300442 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f3eb002ff60 0x7f3eb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:40.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.715+0000 7f3ebd7fa700 1 -- 172.21.15.123:0/1420300442 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3eb4041020 con 0x7f3ec0060280 2024-01-13T11:56:40.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.716+0000 7f3ebffff700 1 --2- 172.21.15.123:0/1420300442 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f3eb002ff60 0x7f3eb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:40.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.716+0000 7f3ebffff700 1 --2- 172.21.15.123:0/1420300442 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f3eb002ff60 0x7f3eb0032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f3eac0096a0 tx=0x7f3eac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:40.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.720+0000 7f3ebd7fa700 1 -- 172.21.15.123:0/1420300442 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3eb4014420 con 0x7f3ec0060280 2024-01-13T11:56:40.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.925+0000 7f3ec70ac700 1 -- 172.21.15.123:0/1420300442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ec01080b0 con 0x7f3ec0060280 2024-01-13T11:56:40.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.925+0000 7f3ebd7fa700 1 -- 172.21.15.123:0/1420300442 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3eb401c920 con 0x7f3ec0060280 2024-01-13T11:56:40.913 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:40.913 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:40.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.927+0000 7f3ec70ac700 1 -- 172.21.15.123:0/1420300442 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f3eb002ff60 msgr2=0x7f3eb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:40.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.927+0000 7f3ec70ac700 1 --2- 172.21.15.123:0/1420300442 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f3eb002ff60 0x7f3eb0032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f3eac0096a0 tx=0x7f3eac006b40).stop 2024-01-13T11:56:40.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.927+0000 7f3ec70ac700 1 -- 172.21.15.123:0/1420300442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ec0060280 msgr2=0x7f3ec01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:40.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.927+0000 7f3ec70ac700 1 --2- 172.21.15.123:0/1420300442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ec0060280 0x7f3ec01066e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f3eb4013510 tx=0x7f3eb400b040).stop 2024-01-13T11:56:40.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.927+0000 7f3ec70ac700 1 -- 172.21.15.123:0/1420300442 shutdown_connections 2024-01-13T11:56:40.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.927+0000 7f3ec70ac700 1 --2- 172.21.15.123:0/1420300442 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f3eb002ff60 0x7f3eb0032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:40.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.927+0000 7f3ec70ac700 1 --2- 172.21.15.123:0/1420300442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ec0060280 0x7f3ec01066e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:40.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.927+0000 7f3ec70ac700 1 -- 172.21.15.123:0/1420300442 >> 172.21.15.123:0/1420300442 conn(0x7f3ec00f0b70 msgr2=0x7f3ec00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:40.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.927+0000 7f3ec70ac700 1 -- 172.21.15.123:0/1420300442 shutdown_connections 2024-01-13T11:56:40.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:40.927+0000 7f3ec70ac700 1 -- 172.21.15.123:0/1420300442 wait complete. 2024-01-13T11:56:40.915 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:41.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:41 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:39.567478+0000 mgr.smithi059.hmyldj (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:41.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:41 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:40.926710+0000 mon.smithi059 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.123:0/1420300442' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:42.293 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:42.294 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:43.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.531+0000 7fce93fff700 1 -- 172.21.15.123:0/1455979563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce940f51a0 msgr2=0x7fce940f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:43.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.531+0000 7fce93fff700 1 --2- 172.21.15.123:0/1455979563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce940f51a0 0x7fce940f55c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fce84009fb0 tx=0x7fce84009b30).stop 2024-01-13T11:56:43.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.532+0000 7fce93fff700 1 -- 172.21.15.123:0/1455979563 shutdown_connections 2024-01-13T11:56:43.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.532+0000 7fce93fff700 1 --2- 172.21.15.123:0/1455979563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce940f51a0 0x7fce940f55c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:43.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.532+0000 7fce93fff700 1 -- 172.21.15.123:0/1455979563 >> 172.21.15.123:0/1455979563 conn(0x7fce940f09a0 msgr2=0x7fce940f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:43.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.532+0000 7fce93fff700 1 -- 172.21.15.123:0/1455979563 shutdown_connections 2024-01-13T11:56:43.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.532+0000 7fce93fff700 1 -- 172.21.15.123:0/1455979563 wait complete. 2024-01-13T11:56:43.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.532+0000 7fce93fff700 1 Processor -- start 2024-01-13T11:56:43.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.532+0000 7fce93fff700 1 -- start start 2024-01-13T11:56:43.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.532+0000 7fce93fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce940f51a0 0x7fce941053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:43.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.532+0000 7fce93fff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce941058a0 con 0x7fce940f51a0 2024-01-13T11:56:43.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.533+0000 7fce92ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce940f51a0 0x7fce941053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:43.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.533+0000 7fce92ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce940f51a0 0x7fce941053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:34820/0 (socket says 172.21.15.123:34820) 2024-01-13T11:56:43.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.533+0000 7fce92ffd700 1 -- 172.21.15.123:0/1118326643 learned_addr learned my addr 172.21.15.123:0/1118326643 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:43.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.533+0000 7fce92ffd700 1 -- 172.21.15.123:0/1118326643 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce84005040 con 0x7fce940f51a0 2024-01-13T11:56:43.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.533+0000 7fce92ffd700 1 --2- 172.21.15.123:0/1118326643 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce940f51a0 0x7fce941053b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fce84013510 tx=0x7fce8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:43.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.533+0000 7fce7b7fe700 1 -- 172.21.15.123:0/1118326643 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce84009320 con 0x7fce940f51a0 2024-01-13T11:56:43.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.533+0000 7fce7b7fe700 1 -- 172.21.15.123:0/1118326643 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce84013f00 con 0x7fce940f51a0 2024-01-13T11:56:43.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.533+0000 7fce7b7fe700 1 -- 172.21.15.123:0/1118326643 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce84014dc0 con 0x7fce940f51a0 2024-01-13T11:56:43.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.533+0000 7fce93fff700 1 -- 172.21.15.123:0/1118326643 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce94105aa0 con 0x7fce940f51a0 2024-01-13T11:56:43.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.533+0000 7fce93fff700 1 -- 172.21.15.123:0/1118326643 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce94105f30 con 0x7fce940f51a0 2024-01-13T11:56:43.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.534+0000 7fce7b7fe700 1 -- 172.21.15.123:0/1118326643 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce840095f0 con 0x7fce940f51a0 2024-01-13T11:56:43.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.534+0000 7fce93fff700 1 -- 172.21.15.123:0/1118326643 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce94045ad0 con 0x7fce940f51a0 2024-01-13T11:56:43.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.534+0000 7fce7b7fe700 1 --2- 172.21.15.123:0/1118326643 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce7c02ff60 0x7fce7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:43.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.534+0000 7fce7b7fe700 1 -- 172.21.15.123:0/1118326643 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce84041100 con 0x7fce940f51a0 2024-01-13T11:56:43.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.535+0000 7fce927fc700 1 --2- 172.21.15.123:0/1118326643 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce7c02ff60 0x7fce7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:43.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.535+0000 7fce927fc700 1 --2- 172.21.15.123:0/1118326643 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce7c02ff60 0x7fce7c032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fce880096a0 tx=0x7fce88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:43.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.539+0000 7fce7b7fe700 1 -- 172.21.15.123:0/1118326643 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce84014420 con 0x7fce940f51a0 2024-01-13T11:56:43.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:43 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:41.567742+0000 mgr.smithi059.hmyldj (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:43.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.744+0000 7fce93fff700 1 -- 172.21.15.123:0/1118326643 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce94106de0 con 0x7fce940f51a0 2024-01-13T11:56:43.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.744+0000 7fce7b7fe700 1 -- 172.21.15.123:0/1118326643 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce84010230 con 0x7fce940f51a0 2024-01-13T11:56:43.732 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:43.733 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:43.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.746+0000 7fce93fff700 1 -- 172.21.15.123:0/1118326643 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce7c02ff60 msgr2=0x7fce7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:43.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.746+0000 7fce93fff700 1 --2- 172.21.15.123:0/1118326643 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce7c02ff60 0x7fce7c032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fce880096a0 tx=0x7fce88006b40).stop 2024-01-13T11:56:43.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.746+0000 7fce93fff700 1 -- 172.21.15.123:0/1118326643 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce940f51a0 msgr2=0x7fce941053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:43.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.746+0000 7fce93fff700 1 --2- 172.21.15.123:0/1118326643 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce940f51a0 0x7fce941053b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fce84013510 tx=0x7fce8400b040).stop 2024-01-13T11:56:43.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.746+0000 7fce93fff700 1 -- 172.21.15.123:0/1118326643 shutdown_connections 2024-01-13T11:56:43.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.746+0000 7fce93fff700 1 --2- 172.21.15.123:0/1118326643 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce7c02ff60 0x7fce7c032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:43.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.746+0000 7fce93fff700 1 --2- 172.21.15.123:0/1118326643 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce940f51a0 0x7fce941053b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:43.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.746+0000 7fce93fff700 1 -- 172.21.15.123:0/1118326643 >> 172.21.15.123:0/1118326643 conn(0x7fce940f09a0 msgr2=0x7fce940f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:43.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.746+0000 7fce93fff700 1 -- 172.21.15.123:0/1118326643 shutdown_connections 2024-01-13T11:56:43.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:43.746+0000 7fce93fff700 1 -- 172.21.15.123:0/1118326643 wait complete. 2024-01-13T11:56:43.735 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:44.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:44 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:43.745882+0000 mon.smithi059 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.123:0/1118326643' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:45.117 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:45.117 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:45.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:45 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:43.568111+0000 mgr.smithi059.hmyldj (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:46.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.285+0000 7ffbe9615700 1 -- 172.21.15.123:0/4071265587 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbe40601e0 msgr2=0x7ffbe40f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:46.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.285+0000 7ffbe9615700 1 --2- 172.21.15.123:0/4071265587 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbe40601e0 0x7ffbe40f9190 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7ffbcc009fb0 tx=0x7ffbcc009b30).stop 2024-01-13T11:56:46.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.285+0000 7ffbe9615700 1 -- 172.21.15.123:0/4071265587 shutdown_connections 2024-01-13T11:56:46.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.285+0000 7ffbe9615700 1 --2- 172.21.15.123:0/4071265587 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbe40601e0 0x7ffbe40f9190 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:46.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.285+0000 7ffbe9615700 1 -- 172.21.15.123:0/4071265587 >> 172.21.15.123:0/4071265587 conn(0x7ffbe40f0b70 msgr2=0x7ffbe40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:46.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.285+0000 7ffbe9615700 1 -- 172.21.15.123:0/4071265587 shutdown_connections 2024-01-13T11:56:46.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.285+0000 7ffbe9615700 1 -- 172.21.15.123:0/4071265587 wait complete. 2024-01-13T11:56:46.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.286+0000 7ffbe9615700 1 Processor -- start 2024-01-13T11:56:46.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.286+0000 7ffbe9615700 1 -- start start 2024-01-13T11:56:46.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.286+0000 7ffbe9615700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbe40601e0 0x7ffbe40f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:46.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.286+0000 7ffbe9615700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe40f5090 con 0x7ffbe40601e0 2024-01-13T11:56:46.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.286+0000 7ffbe2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbe40601e0 0x7ffbe40f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:46.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.286+0000 7ffbe2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbe40601e0 0x7ffbe40f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:34840/0 (socket says 172.21.15.123:34840) 2024-01-13T11:56:46.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.286+0000 7ffbe2ffd700 1 -- 172.21.15.123:0/1110251046 learned_addr learned my addr 172.21.15.123:0/1110251046 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:46.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.286+0000 7ffbe2ffd700 1 -- 172.21.15.123:0/1110251046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc005040 con 0x7ffbe40601e0 2024-01-13T11:56:46.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.286+0000 7ffbe2ffd700 1 --2- 172.21.15.123:0/1110251046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbe40601e0 0x7ffbe40f6c10 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7ffbcc004dd0 tx=0x7ffbcc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:46.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.287+0000 7ffbdb7fe700 1 -- 172.21.15.123:0/1110251046 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffbcc018070 con 0x7ffbe40601e0 2024-01-13T11:56:46.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.287+0000 7ffbe9615700 1 -- 172.21.15.123:0/1110251046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe40f5230 con 0x7ffbe40601e0 2024-01-13T11:56:46.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.287+0000 7ffbdb7fe700 1 -- 172.21.15.123:0/1110251046 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffbcc013410 con 0x7ffbe40601e0 2024-01-13T11:56:46.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.287+0000 7ffbe9615700 1 -- 172.21.15.123:0/1110251046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe40f56c0 con 0x7ffbe40601e0 2024-01-13T11:56:46.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.287+0000 7ffbdb7fe700 1 -- 172.21.15.123:0/1110251046 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffbcc01c3c0 con 0x7ffbe40601e0 2024-01-13T11:56:46.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.287+0000 7ffbdb7fe700 1 -- 172.21.15.123:0/1110251046 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffbcc01d420 con 0x7ffbe40601e0 2024-01-13T11:56:46.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.288+0000 7ffbe9615700 1 -- 172.21.15.123:0/1110251046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe4059100 con 0x7ffbe40601e0 2024-01-13T11:56:46.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.288+0000 7ffbdb7fe700 1 --2- 172.21.15.123:0/1110251046 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffbd002ff60 0x7ffbd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:46.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.288+0000 7ffbdb7fe700 1 -- 172.21.15.123:0/1110251046 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffbcc0413d0 con 0x7ffbe40601e0 2024-01-13T11:56:46.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.288+0000 7ffbe27fc700 1 --2- 172.21.15.123:0/1110251046 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffbd002ff60 0x7ffbd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:46.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.288+0000 7ffbe27fc700 1 --2- 172.21.15.123:0/1110251046 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffbd002ff60 0x7ffbd0032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7ffbd40096a0 tx=0x7ffbd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:46.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.293+0000 7ffbdb7fe700 1 -- 172.21.15.123:0/1110251046 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffbcc01d820 con 0x7ffbe40601e0 2024-01-13T11:56:46.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.501+0000 7ffbe9615700 1 -- 172.21.15.123:0/1110251046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffbe4045ad0 con 0x7ffbe40601e0 2024-01-13T11:56:46.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.502+0000 7ffbdb7fe700 1 -- 172.21.15.123:0/1110251046 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffbcc01b370 con 0x7ffbe40601e0 2024-01-13T11:56:46.490 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:46.490 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:46.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.504+0000 7ffbe9615700 1 -- 172.21.15.123:0/1110251046 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffbd002ff60 msgr2=0x7ffbd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:46.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.504+0000 7ffbe9615700 1 --2- 172.21.15.123:0/1110251046 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffbd002ff60 0x7ffbd0032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7ffbd40096a0 tx=0x7ffbd4006b40).stop 2024-01-13T11:56:46.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.504+0000 7ffbe9615700 1 -- 172.21.15.123:0/1110251046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbe40601e0 msgr2=0x7ffbe40f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:46.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.504+0000 7ffbe9615700 1 --2- 172.21.15.123:0/1110251046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbe40601e0 0x7ffbe40f6c10 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7ffbcc004dd0 tx=0x7ffbcc009b30).stop 2024-01-13T11:56:46.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.504+0000 7ffbe9615700 1 -- 172.21.15.123:0/1110251046 shutdown_connections 2024-01-13T11:56:46.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.504+0000 7ffbe9615700 1 --2- 172.21.15.123:0/1110251046 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ffbd002ff60 0x7ffbd0032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:46.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.504+0000 7ffbe9615700 1 --2- 172.21.15.123:0/1110251046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbe40601e0 0x7ffbe40f6c10 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:46.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.504+0000 7ffbe9615700 1 -- 172.21.15.123:0/1110251046 >> 172.21.15.123:0/1110251046 conn(0x7ffbe40f0b70 msgr2=0x7ffbe40f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:46.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.504+0000 7ffbe9615700 1 -- 172.21.15.123:0/1110251046 shutdown_connections 2024-01-13T11:56:46.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:46.504+0000 7ffbe9615700 1 -- 172.21.15.123:0/1110251046 wait complete. 2024-01-13T11:56:46.495 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:47.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:47 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:45.568537+0000 mgr.smithi059.hmyldj (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:47.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:47 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:46.503544+0000 mon.smithi059 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.123:0/1110251046' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:47.879 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:47.879 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:48.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.981+0000 7f27d1215700 1 -- 172.21.15.123:0/633327723 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27cc0f78e0 msgr2=0x7f27cc0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:48.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.981+0000 7f27d1215700 1 --2- 172.21.15.123:0/633327723 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27cc0f78e0 0x7f27cc0f7d00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f27bc009fb0 tx=0x7f27bc009b30).stop 2024-01-13T11:56:48.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.981+0000 7f27d1215700 1 -- 172.21.15.123:0/633327723 shutdown_connections 2024-01-13T11:56:48.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.981+0000 7f27d1215700 1 --2- 172.21.15.123:0/633327723 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27cc0f78e0 0x7f27cc0f7d00 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:48.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.981+0000 7f27d1215700 1 -- 172.21.15.123:0/633327723 >> 172.21.15.123:0/633327723 conn(0x7f27cc0f0120 msgr2=0x7f27cc0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:48.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.981+0000 7f27d1215700 1 -- 172.21.15.123:0/633327723 shutdown_connections 2024-01-13T11:56:48.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.981+0000 7f27d1215700 1 -- 172.21.15.123:0/633327723 wait complete. 2024-01-13T11:56:48.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.981+0000 7f27d1215700 1 Processor -- start 2024-01-13T11:56:48.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.981+0000 7f27d1215700 1 -- start start 2024-01-13T11:56:48.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.981+0000 7f27d1215700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27cc0f78e0 0x7f27cc1086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:48.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.982+0000 7f27d1215700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc108bc0 con 0x7f27cc0f78e0 2024-01-13T11:56:48.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.982+0000 7f27cbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27cc0f78e0 0x7f27cc1086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:48.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.982+0000 7f27cbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27cc0f78e0 0x7f27cc1086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:59662/0 (socket says 172.21.15.123:59662) 2024-01-13T11:56:48.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.982+0000 7f27cbfff700 1 -- 172.21.15.123:0/841843350 learned_addr learned my addr 172.21.15.123:0/841843350 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:48.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.982+0000 7f27cbfff700 1 -- 172.21.15.123:0/841843350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc005040 con 0x7f27cc0f78e0 2024-01-13T11:56:48.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.982+0000 7f27cbfff700 1 --2- 172.21.15.123:0/841843350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27cc0f78e0 0x7f27cc1086d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f27bc013510 tx=0x7f27bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:48.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.982+0000 7f27c8ff9700 1 -- 172.21.15.123:0/841843350 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27bc009320 con 0x7f27cc0f78e0 2024-01-13T11:56:48.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.982+0000 7f27d1215700 1 -- 172.21.15.123:0/841843350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27cc108dc0 con 0x7f27cc0f78e0 2024-01-13T11:56:48.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.982+0000 7f27c8ff9700 1 -- 172.21.15.123:0/841843350 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27bc013f00 con 0x7f27cc0f78e0 2024-01-13T11:56:48.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.982+0000 7f27c8ff9700 1 -- 172.21.15.123:0/841843350 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27bc014dc0 con 0x7f27cc0f78e0 2024-01-13T11:56:48.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.982+0000 7f27d1215700 1 -- 172.21.15.123:0/841843350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27cc109250 con 0x7f27cc0f78e0 2024-01-13T11:56:48.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.983+0000 7f27c8ff9700 1 -- 172.21.15.123:0/841843350 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f27bc0095f0 con 0x7f27cc0f78e0 2024-01-13T11:56:48.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.983+0000 7f27d1215700 1 -- 172.21.15.123:0/841843350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27cc045ad0 con 0x7f27cc0f78e0 2024-01-13T11:56:48.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.983+0000 7f27c8ff9700 1 --2- 172.21.15.123:0/841843350 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f27b402ff60 0x7f27b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:48.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.983+0000 7f27c8ff9700 1 -- 172.21.15.123:0/841843350 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f27bc041040 con 0x7f27cc0f78e0 2024-01-13T11:56:48.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.984+0000 7f27cb7fe700 1 --2- 172.21.15.123:0/841843350 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f27b402ff60 0x7f27b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:48.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.984+0000 7f27cb7fe700 1 --2- 172.21.15.123:0/841843350 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f27b402ff60 0x7f27b4032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f27c00096a0 tx=0x7f27c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:48.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:48.988+0000 7f27c8ff9700 1 -- 172.21.15.123:0/841843350 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27bc014420 con 0x7f27cc0f78e0 2024-01-13T11:56:49.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:49.192+0000 7f27d1215700 1 -- 172.21.15.123:0/841843350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f27cc10a100 con 0x7f27cc0f78e0 2024-01-13T11:56:49.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:49.193+0000 7f27c8ff9700 1 -- 172.21.15.123:0/841843350 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f27bc010230 con 0x7f27cc0f78e0 2024-01-13T11:56:49.180 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:49.180 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:49.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:49.194+0000 7f27d1215700 1 -- 172.21.15.123:0/841843350 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f27b402ff60 msgr2=0x7f27b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:49.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:49.194+0000 7f27d1215700 1 --2- 172.21.15.123:0/841843350 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f27b402ff60 0x7f27b4032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f27c00096a0 tx=0x7f27c0006b40).stop 2024-01-13T11:56:49.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:49.195+0000 7f27d1215700 1 -- 172.21.15.123:0/841843350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27cc0f78e0 msgr2=0x7f27cc1086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:49.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:49.195+0000 7f27d1215700 1 --2- 172.21.15.123:0/841843350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27cc0f78e0 0x7f27cc1086d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f27bc013510 tx=0x7f27bc00b040).stop 2024-01-13T11:56:49.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:49.195+0000 7f27d1215700 1 -- 172.21.15.123:0/841843350 shutdown_connections 2024-01-13T11:56:49.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:49.195+0000 7f27d1215700 1 --2- 172.21.15.123:0/841843350 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f27b402ff60 0x7f27b4032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:49.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:49.195+0000 7f27d1215700 1 --2- 172.21.15.123:0/841843350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27cc0f78e0 0x7f27cc1086d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:49.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:49.195+0000 7f27d1215700 1 -- 172.21.15.123:0/841843350 >> 172.21.15.123:0/841843350 conn(0x7f27cc0f0120 msgr2=0x7f27cc0f0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:49.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:49.195+0000 7f27d1215700 1 -- 172.21.15.123:0/841843350 shutdown_connections 2024-01-13T11:56:49.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:49.195+0000 7f27d1215700 1 -- 172.21.15.123:0/841843350 wait complete. 2024-01-13T11:56:49.182 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:49.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:49 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:47.568860+0000 mgr.smithi059.hmyldj (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:49.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:49 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:49.193332+0000 mon.smithi059 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.123:0/841843350' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:50.580 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:50.583 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:51.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:49.569175+0000 mgr.smithi059.hmyldj (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:51.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.764+0000 7f4fa9050700 1 -- 172.21.15.123:0/2174866920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fa40f5410 msgr2=0x7f4fa40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:51.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.764+0000 7f4fa9050700 1 --2- 172.21.15.123:0/2174866920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fa40f5410 0x7f4fa40f5830 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f4f98009fb0 tx=0x7f4f98009b30).stop 2024-01-13T11:56:51.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.764+0000 7f4fa9050700 1 -- 172.21.15.123:0/2174866920 shutdown_connections 2024-01-13T11:56:51.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.764+0000 7f4fa9050700 1 --2- 172.21.15.123:0/2174866920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fa40f5410 0x7f4fa40f5830 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:51.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.764+0000 7f4fa9050700 1 -- 172.21.15.123:0/2174866920 >> 172.21.15.123:0/2174866920 conn(0x7f4fa40f0b70 msgr2=0x7f4fa40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:51.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.764+0000 7f4fa9050700 1 -- 172.21.15.123:0/2174866920 shutdown_connections 2024-01-13T11:56:51.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.764+0000 7f4fa9050700 1 -- 172.21.15.123:0/2174866920 wait complete. 2024-01-13T11:56:51.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.765+0000 7f4fa9050700 1 Processor -- start 2024-01-13T11:56:51.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.765+0000 7f4fa9050700 1 -- start start 2024-01-13T11:56:51.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.765+0000 7f4fa9050700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fa40f5410 0x7f4fa41055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:51.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.765+0000 7f4fa9050700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa4105ac0 con 0x7f4fa40f5410 2024-01-13T11:56:51.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.766+0000 7f4fa2d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fa40f5410 0x7f4fa41055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:51.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.766+0000 7f4fa2d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fa40f5410 0x7f4fa41055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:59674/0 (socket says 172.21.15.123:59674) 2024-01-13T11:56:51.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.766+0000 7f4fa2d9d700 1 -- 172.21.15.123:0/1537608360 learned_addr learned my addr 172.21.15.123:0/1537608360 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:51.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.766+0000 7f4fa2d9d700 1 -- 172.21.15.123:0/1537608360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f98005040 con 0x7f4fa40f5410 2024-01-13T11:56:51.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.766+0000 7f4fa2d9d700 1 --2- 172.21.15.123:0/1537608360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fa40f5410 0x7f4fa41055d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f4f98014000 tx=0x7f4f9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:51.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.766+0000 7f4f8b7fe700 1 -- 172.21.15.123:0/1537608360 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f980095f0 con 0x7f4fa40f5410 2024-01-13T11:56:51.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.766+0000 7f4fa9050700 1 -- 172.21.15.123:0/1537608360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa4105cc0 con 0x7f4fa40f5410 2024-01-13T11:56:51.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.766+0000 7f4f8b7fe700 1 -- 172.21.15.123:0/1537608360 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f98007e70 con 0x7f4fa40f5410 2024-01-13T11:56:51.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.766+0000 7f4f8b7fe700 1 -- 172.21.15.123:0/1537608360 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f98014ea0 con 0x7f4fa40f5410 2024-01-13T11:56:51.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.766+0000 7f4fa9050700 1 -- 172.21.15.123:0/1537608360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa4106150 con 0x7f4fa40f5410 2024-01-13T11:56:51.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.767+0000 7f4fa9050700 1 -- 172.21.15.123:0/1537608360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa4045ad0 con 0x7f4fa40f5410 2024-01-13T11:56:51.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.767+0000 7f4f8b7fe700 1 -- 172.21.15.123:0/1537608360 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4f98025070 con 0x7f4fa40f5410 2024-01-13T11:56:51.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.767+0000 7f4f8b7fe700 1 --2- 172.21.15.123:0/1537608360 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f4f8c02ff60 0x7f4f8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:51.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.768+0000 7f4f8b7fe700 1 -- 172.21.15.123:0/1537608360 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4f98040a00 con 0x7f4fa40f5410 2024-01-13T11:56:51.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.768+0000 7f4fa259c700 1 --2- 172.21.15.123:0/1537608360 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f4f8c02ff60 0x7f4f8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:51.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.768+0000 7f4fa259c700 1 --2- 172.21.15.123:0/1537608360 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f4f8c02ff60 0x7f4f8c032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f4f940096a0 tx=0x7f4f94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:51.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.772+0000 7f4f8b7fe700 1 -- 172.21.15.123:0/1537608360 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4f980079c0 con 0x7f4fa40f5410 2024-01-13T11:56:51.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.978+0000 7f4fa9050700 1 -- 172.21.15.123:0/1537608360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4fa4106fa0 con 0x7f4fa40f5410 2024-01-13T11:56:51.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.979+0000 7f4f8b7fe700 1 -- 172.21.15.123:0/1537608360 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4f9800f300 con 0x7f4fa40f5410 2024-01-13T11:56:51.966 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:51.966 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:51.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.981+0000 7f4fa9050700 1 -- 172.21.15.123:0/1537608360 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f4f8c02ff60 msgr2=0x7f4f8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:51.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.981+0000 7f4fa9050700 1 --2- 172.21.15.123:0/1537608360 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f4f8c02ff60 0x7f4f8c032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f4f940096a0 tx=0x7f4f94006b40).stop 2024-01-13T11:56:51.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.981+0000 7f4fa9050700 1 -- 172.21.15.123:0/1537608360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fa40f5410 msgr2=0x7f4fa41055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:51.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.981+0000 7f4fa9050700 1 --2- 172.21.15.123:0/1537608360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fa40f5410 0x7f4fa41055d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f4f98014000 tx=0x7f4f9800b040).stop 2024-01-13T11:56:51.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.981+0000 7f4fa9050700 1 -- 172.21.15.123:0/1537608360 shutdown_connections 2024-01-13T11:56:51.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.981+0000 7f4fa9050700 1 --2- 172.21.15.123:0/1537608360 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f4f8c02ff60 0x7f4f8c032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:51.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.981+0000 7f4fa9050700 1 --2- 172.21.15.123:0/1537608360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fa40f5410 0x7f4fa41055d0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:51.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.981+0000 7f4fa9050700 1 -- 172.21.15.123:0/1537608360 >> 172.21.15.123:0/1537608360 conn(0x7f4fa40f0b70 msgr2=0x7f4fa40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:51.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.981+0000 7f4fa9050700 1 -- 172.21.15.123:0/1537608360 shutdown_connections 2024-01-13T11:56:51.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:51.981+0000 7f4fa9050700 1 -- 172.21.15.123:0/1537608360 wait complete. 2024-01-13T11:56:51.969 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:52.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:52 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:51.979409+0000 mon.smithi059 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.123:0/1537608360' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:53.371 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:53.372 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:53.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:53 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:51.569411+0000 mgr.smithi059.hmyldj (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:54.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.629+0000 7f6c19ac3700 1 -- 172.21.15.123:0/1422322768 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c140f5410 msgr2=0x7f6c140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:54.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.629+0000 7f6c19ac3700 1 --2- 172.21.15.123:0/1422322768 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c140f5410 0x7f6c140f5830 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f6c04009fb0 tx=0x7f6c04009b30).stop 2024-01-13T11:56:54.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.629+0000 7f6c19ac3700 1 -- 172.21.15.123:0/1422322768 shutdown_connections 2024-01-13T11:56:54.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.629+0000 7f6c19ac3700 1 --2- 172.21.15.123:0/1422322768 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c140f5410 0x7f6c140f5830 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:54.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.629+0000 7f6c19ac3700 1 -- 172.21.15.123:0/1422322768 >> 172.21.15.123:0/1422322768 conn(0x7f6c140f0b70 msgr2=0x7f6c140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:54.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.630+0000 7f6c19ac3700 1 -- 172.21.15.123:0/1422322768 shutdown_connections 2024-01-13T11:56:54.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.630+0000 7f6c19ac3700 1 -- 172.21.15.123:0/1422322768 wait complete. 2024-01-13T11:56:54.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.630+0000 7f6c19ac3700 1 Processor -- start 2024-01-13T11:56:54.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.630+0000 7f6c19ac3700 1 -- start start 2024-01-13T11:56:54.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.630+0000 7f6c19ac3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c140f5410 0x7f6c140fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:54.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.630+0000 7f6c19ac3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c140fb820 con 0x7f6c140f5410 2024-01-13T11:56:54.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.631+0000 7f6c137fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c140f5410 0x7f6c140fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:54.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.631+0000 7f6c137fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c140f5410 0x7f6c140fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:59690/0 (socket says 172.21.15.123:59690) 2024-01-13T11:56:54.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.631+0000 7f6c137fe700 1 -- 172.21.15.123:0/2390483301 learned_addr learned my addr 172.21.15.123:0/2390483301 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:54.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.631+0000 7f6c137fe700 1 -- 172.21.15.123:0/2390483301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c04005040 con 0x7f6c140f5410 2024-01-13T11:56:54.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.631+0000 7f6c137fe700 1 --2- 172.21.15.123:0/2390483301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c140f5410 0x7f6c140fd3a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f6c04013510 tx=0x7f6c0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:54.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.631+0000 7f6bfbfff700 1 -- 172.21.15.123:0/2390483301 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c04009320 con 0x7f6c140f5410 2024-01-13T11:56:54.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.631+0000 7f6c19ac3700 1 -- 172.21.15.123:0/2390483301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c140fb9c0 con 0x7f6c140f5410 2024-01-13T11:56:54.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.631+0000 7f6bfbfff700 1 -- 172.21.15.123:0/2390483301 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c04013f00 con 0x7f6c140f5410 2024-01-13T11:56:54.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.631+0000 7f6bfbfff700 1 -- 172.21.15.123:0/2390483301 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c04014d90 con 0x7f6c140f5410 2024-01-13T11:56:54.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.631+0000 7f6c19ac3700 1 -- 172.21.15.123:0/2390483301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c140fbe50 con 0x7f6c140f5410 2024-01-13T11:56:54.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.632+0000 7f6bfbfff700 1 -- 172.21.15.123:0/2390483301 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c040263a0 con 0x7f6c140f5410 2024-01-13T11:56:54.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.632+0000 7f6bfbfff700 1 --2- 172.21.15.123:0/2390483301 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6bfc02ffb0 0x7f6bfc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:54.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.632+0000 7f6bfbfff700 1 -- 172.21.15.123:0/2390483301 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c040410c0 con 0x7f6c140f5410 2024-01-13T11:56:54.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.632+0000 7f6c19ac3700 1 -- 172.21.15.123:0/2390483301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c14059100 con 0x7f6c140f5410 2024-01-13T11:56:54.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.632+0000 7f6c12ffd700 1 --2- 172.21.15.123:0/2390483301 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6bfc02ffb0 0x7f6bfc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:54.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.633+0000 7f6c12ffd700 1 --2- 172.21.15.123:0/2390483301 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6bfc02ffb0 0x7f6bfc032470 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f6c080096a0 tx=0x7f6c08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:54.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.637+0000 7f6bfbfff700 1 -- 172.21.15.123:0/2390483301 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c04012920 con 0x7f6c140f5410 2024-01-13T11:56:54.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.841+0000 7f6c19ac3700 1 -- 172.21.15.123:0/2390483301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c140fc550 con 0x7f6c140f5410 2024-01-13T11:56:54.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.841+0000 7f6bfbfff700 1 -- 172.21.15.123:0/2390483301 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6c0402b020 con 0x7f6c140f5410 2024-01-13T11:56:54.829 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:54.829 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:54.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.843+0000 7f6c19ac3700 1 -- 172.21.15.123:0/2390483301 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6bfc02ffb0 msgr2=0x7f6bfc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:54.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.843+0000 7f6c19ac3700 1 --2- 172.21.15.123:0/2390483301 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6bfc02ffb0 0x7f6bfc032470 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f6c080096a0 tx=0x7f6c08006b40).stop 2024-01-13T11:56:54.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.843+0000 7f6c19ac3700 1 -- 172.21.15.123:0/2390483301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c140f5410 msgr2=0x7f6c140fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:54.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.843+0000 7f6c19ac3700 1 --2- 172.21.15.123:0/2390483301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c140f5410 0x7f6c140fd3a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f6c04013510 tx=0x7f6c0400b040).stop 2024-01-13T11:56:54.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.843+0000 7f6c19ac3700 1 -- 172.21.15.123:0/2390483301 shutdown_connections 2024-01-13T11:56:54.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.843+0000 7f6c19ac3700 1 --2- 172.21.15.123:0/2390483301 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6bfc02ffb0 0x7f6bfc032470 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:54.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.843+0000 7f6c19ac3700 1 --2- 172.21.15.123:0/2390483301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c140f5410 0x7f6c140fd3a0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:54.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.843+0000 7f6c19ac3700 1 -- 172.21.15.123:0/2390483301 >> 172.21.15.123:0/2390483301 conn(0x7f6c140f0b70 msgr2=0x7f6c1405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:54.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.843+0000 7f6c19ac3700 1 -- 172.21.15.123:0/2390483301 shutdown_connections 2024-01-13T11:56:54.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:54.843+0000 7f6c19ac3700 1 -- 172.21.15.123:0/2390483301 wait complete. 2024-01-13T11:56:54.832 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:55.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:55 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:53.569633+0000 mgr.smithi059.hmyldj (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:55.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:55 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:54.842914+0000 mon.smithi059 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.123:0/2390483301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:56.228 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:56.229 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:57.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.359+0000 7fa05277c700 1 -- 172.21.15.123:0/4190197089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0f5410 msgr2=0x7fa04c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:57.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.359+0000 7fa05277c700 1 --2- 172.21.15.123:0/4190197089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0f5410 0x7fa04c0f5830 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fa040009fb0 tx=0x7fa040009b30).stop 2024-01-13T11:56:57.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.359+0000 7fa05277c700 1 -- 172.21.15.123:0/4190197089 shutdown_connections 2024-01-13T11:56:57.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.359+0000 7fa05277c700 1 --2- 172.21.15.123:0/4190197089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0f5410 0x7fa04c0f5830 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:57.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.359+0000 7fa05277c700 1 -- 172.21.15.123:0/4190197089 >> 172.21.15.123:0/4190197089 conn(0x7fa04c0f0b70 msgr2=0x7fa04c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:57.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.359+0000 7fa05277c700 1 -- 172.21.15.123:0/4190197089 shutdown_connections 2024-01-13T11:56:57.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.359+0000 7fa05277c700 1 -- 172.21.15.123:0/4190197089 wait complete. 2024-01-13T11:56:57.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.360+0000 7fa05277c700 1 Processor -- start 2024-01-13T11:56:57.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.360+0000 7fa05277c700 1 -- start start 2024-01-13T11:56:57.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.360+0000 7fa05277c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0f5410 0x7fa04c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:57.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.360+0000 7fa05277c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c106c10 con 0x7fa04c0f5410 2024-01-13T11:56:57.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.360+0000 7fa04bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0f5410 0x7fa04c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:57.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.360+0000 7fa04bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0f5410 0x7fa04c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:44514/0 (socket says 172.21.15.123:44514) 2024-01-13T11:56:57.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.360+0000 7fa04bfff700 1 -- 172.21.15.123:0/3161340812 learned_addr learned my addr 172.21.15.123:0/3161340812 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:56:57.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.361+0000 7fa04bfff700 1 -- 172.21.15.123:0/3161340812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa040005040 con 0x7fa04c0f5410 2024-01-13T11:56:57.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.361+0000 7fa04bfff700 1 --2- 172.21.15.123:0/3161340812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0f5410 0x7fa04c106720 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fa040014000 tx=0x7fa04000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:57.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.361+0000 7fa048ff9700 1 -- 172.21.15.123:0/3161340812 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0400095f0 con 0x7fa04c0f5410 2024-01-13T11:56:57.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.361+0000 7fa05277c700 1 -- 172.21.15.123:0/3161340812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04c106e10 con 0x7fa04c0f5410 2024-01-13T11:56:57.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.361+0000 7fa048ff9700 1 -- 172.21.15.123:0/3161340812 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa040007e70 con 0x7fa04c0f5410 2024-01-13T11:56:57.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.361+0000 7fa048ff9700 1 -- 172.21.15.123:0/3161340812 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa040014ea0 con 0x7fa04c0f5410 2024-01-13T11:56:57.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.361+0000 7fa05277c700 1 -- 172.21.15.123:0/3161340812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04c1072a0 con 0x7fa04c0f5410 2024-01-13T11:56:57.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.362+0000 7fa048ff9700 1 -- 172.21.15.123:0/3161340812 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa0400079c0 con 0x7fa04c0f5410 2024-01-13T11:56:57.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.362+0000 7fa05277c700 1 -- 172.21.15.123:0/3161340812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa04c045ad0 con 0x7fa04c0f5410 2024-01-13T11:56:57.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.362+0000 7fa048ff9700 1 --2- 172.21.15.123:0/3161340812 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa03402ff60 0x7fa034032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:56:57.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.362+0000 7fa048ff9700 1 -- 172.21.15.123:0/3161340812 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa0400408e0 con 0x7fa04c0f5410 2024-01-13T11:56:57.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.362+0000 7fa04b7fe700 1 --2- 172.21.15.123:0/3161340812 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa03402ff60 0x7fa034032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:56:57.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.363+0000 7fa04b7fe700 1 --2- 172.21.15.123:0/3161340812 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa03402ff60 0x7fa034032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fa03c0096a0 tx=0x7fa03c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:56:57.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.367+0000 7fa048ff9700 1 -- 172.21.15.123:0/3161340812 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa04000f950 con 0x7fa04c0f5410 2024-01-13T11:56:57.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.571+0000 7fa05277c700 1 -- 172.21.15.123:0/3161340812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa04c108150 con 0x7fa04c0f5410 2024-01-13T11:56:57.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.571+0000 7fa048ff9700 1 -- 172.21.15.123:0/3161340812 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa040010230 con 0x7fa04c0f5410 2024-01-13T11:56:57.559 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:56:57.559 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:56:57.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.573+0000 7fa05277c700 1 -- 172.21.15.123:0/3161340812 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa03402ff60 msgr2=0x7fa034032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:57.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.573+0000 7fa05277c700 1 --2- 172.21.15.123:0/3161340812 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa03402ff60 0x7fa034032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fa03c0096a0 tx=0x7fa03c006b40).stop 2024-01-13T11:56:57.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.573+0000 7fa05277c700 1 -- 172.21.15.123:0/3161340812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0f5410 msgr2=0x7fa04c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:56:57.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.573+0000 7fa05277c700 1 --2- 172.21.15.123:0/3161340812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0f5410 0x7fa04c106720 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fa040014000 tx=0x7fa04000b040).stop 2024-01-13T11:56:57.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.573+0000 7fa05277c700 1 -- 172.21.15.123:0/3161340812 shutdown_connections 2024-01-13T11:56:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.573+0000 7fa05277c700 1 --2- 172.21.15.123:0/3161340812 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa03402ff60 0x7fa034032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.573+0000 7fa05277c700 1 --2- 172.21.15.123:0/3161340812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa04c0f5410 0x7fa04c106720 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:56:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.573+0000 7fa05277c700 1 -- 172.21.15.123:0/3161340812 >> 172.21.15.123:0/3161340812 conn(0x7fa04c0f0b70 msgr2=0x7fa04c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:56:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.573+0000 7fa05277c700 1 -- 172.21.15.123:0/3161340812 shutdown_connections 2024-01-13T11:56:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:56:57.573+0000 7fa05277c700 1 -- 172.21.15.123:0/3161340812 wait complete. 2024-01-13T11:56:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:56:57.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:55.569900+0000 mgr.smithi059.hmyldj (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:58.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:58 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:56:57.572612+0000 mon.smithi059 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.123:0/3161340812' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:56:58.967 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:56:58.968 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:56:59.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:56:59 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:57.570181+0000 mgr.smithi059.hmyldj (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:00.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.196+0000 7f9cb23bd700 1 -- 172.21.15.123:0/2423844828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cac060280 msgr2=0x7f9cac0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:00.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.196+0000 7f9cb23bd700 1 --2- 172.21.15.123:0/2423844828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cac060280 0x7f9cac0f90f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f9c9c009fb0 tx=0x7f9c9c009b30).stop 2024-01-13T11:57:00.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.196+0000 7f9cb23bd700 1 -- 172.21.15.123:0/2423844828 shutdown_connections 2024-01-13T11:57:00.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.196+0000 7f9cb23bd700 1 --2- 172.21.15.123:0/2423844828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cac060280 0x7f9cac0f90f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:00.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.196+0000 7f9cb23bd700 1 -- 172.21.15.123:0/2423844828 >> 172.21.15.123:0/2423844828 conn(0x7f9cac0f0b70 msgr2=0x7f9cac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:00.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.196+0000 7f9cb23bd700 1 -- 172.21.15.123:0/2423844828 shutdown_connections 2024-01-13T11:57:00.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.196+0000 7f9cb23bd700 1 -- 172.21.15.123:0/2423844828 wait complete. 2024-01-13T11:57:00.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.197+0000 7f9cb23bd700 1 Processor -- start 2024-01-13T11:57:00.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.197+0000 7f9cb23bd700 1 -- start start 2024-01-13T11:57:00.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.197+0000 7f9cb23bd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cac060280 0x7f9cac1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:00.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.197+0000 7f9cb23bd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac106bd0 con 0x7f9cac060280 2024-01-13T11:57:00.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.197+0000 7f9cabfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cac060280 0x7f9cac1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:00.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.197+0000 7f9cabfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cac060280 0x7f9cac1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:44534/0 (socket says 172.21.15.123:44534) 2024-01-13T11:57:00.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.197+0000 7f9cabfff700 1 -- 172.21.15.123:0/4176617177 learned_addr learned my addr 172.21.15.123:0/4176617177 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:00.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.198+0000 7f9cabfff700 1 -- 172.21.15.123:0/4176617177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c005040 con 0x7f9cac060280 2024-01-13T11:57:00.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.198+0000 7f9cabfff700 1 --2- 172.21.15.123:0/4176617177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cac060280 0x7f9cac1066e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f9c9c013510 tx=0x7f9c9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:00.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.198+0000 7f9ca8ff9700 1 -- 172.21.15.123:0/4176617177 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c9c009320 con 0x7f9cac060280 2024-01-13T11:57:00.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.198+0000 7f9cb23bd700 1 -- 172.21.15.123:0/4176617177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cac106dd0 con 0x7f9cac060280 2024-01-13T11:57:00.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.198+0000 7f9ca8ff9700 1 -- 172.21.15.123:0/4176617177 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c9c013f00 con 0x7f9cac060280 2024-01-13T11:57:00.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.198+0000 7f9ca8ff9700 1 -- 172.21.15.123:0/4176617177 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c9c014dc0 con 0x7f9cac060280 2024-01-13T11:57:00.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.198+0000 7f9cb23bd700 1 -- 172.21.15.123:0/4176617177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cac107260 con 0x7f9cac060280 2024-01-13T11:57:00.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.199+0000 7f9ca8ff9700 1 -- 172.21.15.123:0/4176617177 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c9c01d3d0 con 0x7f9cac060280 2024-01-13T11:57:00.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.199+0000 7f9ca8ff9700 1 --2- 172.21.15.123:0/4176617177 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9c9802ff60 0x7f9c98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:00.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.199+0000 7f9ca8ff9700 1 -- 172.21.15.123:0/4176617177 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c9c014640 con 0x7f9cac060280 2024-01-13T11:57:00.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.199+0000 7f9cb23bd700 1 -- 172.21.15.123:0/4176617177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cac059100 con 0x7f9cac060280 2024-01-13T11:57:00.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.199+0000 7f9cab7fe700 1 --2- 172.21.15.123:0/4176617177 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9c9802ff60 0x7f9c98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:00.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.200+0000 7f9cab7fe700 1 --2- 172.21.15.123:0/4176617177 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9c9802ff60 0x7f9c98032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f9c940096a0 tx=0x7f9c94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:00.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.204+0000 7f9ca8ff9700 1 -- 172.21.15.123:0/4176617177 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c9c012920 con 0x7f9cac060280 2024-01-13T11:57:00.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.408+0000 7f9cb23bd700 1 -- 172.21.15.123:0/4176617177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9cac0fbf50 con 0x7f9cac060280 2024-01-13T11:57:00.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.408+0000 7f9ca8ff9700 1 -- 172.21.15.123:0/4176617177 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9c9c012330 con 0x7f9cac060280 2024-01-13T11:57:00.396 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:00.396 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:00.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.410+0000 7f9cb23bd700 1 -- 172.21.15.123:0/4176617177 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9c9802ff60 msgr2=0x7f9c98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:00.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.410+0000 7f9cb23bd700 1 --2- 172.21.15.123:0/4176617177 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9c9802ff60 0x7f9c98032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f9c940096a0 tx=0x7f9c94006b40).stop 2024-01-13T11:57:00.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.410+0000 7f9cb23bd700 1 -- 172.21.15.123:0/4176617177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cac060280 msgr2=0x7f9cac1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:00.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.410+0000 7f9cb23bd700 1 --2- 172.21.15.123:0/4176617177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cac060280 0x7f9cac1066e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f9c9c013510 tx=0x7f9c9c00b040).stop 2024-01-13T11:57:00.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.410+0000 7f9cb23bd700 1 -- 172.21.15.123:0/4176617177 shutdown_connections 2024-01-13T11:57:00.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.410+0000 7f9cb23bd700 1 --2- 172.21.15.123:0/4176617177 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9c9802ff60 0x7f9c98032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:00.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.410+0000 7f9cb23bd700 1 --2- 172.21.15.123:0/4176617177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cac060280 0x7f9cac1066e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:00.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.410+0000 7f9cb23bd700 1 -- 172.21.15.123:0/4176617177 >> 172.21.15.123:0/4176617177 conn(0x7f9cac0f0b70 msgr2=0x7f9cac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:00.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.410+0000 7f9cb23bd700 1 -- 172.21.15.123:0/4176617177 shutdown_connections 2024-01-13T11:57:00.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:00.410+0000 7f9cb23bd700 1 -- 172.21.15.123:0/4176617177 wait complete. 2024-01-13T11:57:00.399 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:01.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:01 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:56:59.570546+0000 mgr.smithi059.hmyldj (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:01.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:01 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:00.409779+0000 mon.smithi059 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.123:0/4176617177' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:01.790 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:01.791 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:02.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.993+0000 7fa635f1e700 1 -- 172.21.15.123:0/94480548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6300601c0 msgr2=0x7fa6300f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:02.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.993+0000 7fa635f1e700 1 --2- 172.21.15.123:0/94480548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6300601c0 0x7fa6300f91b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fa620009fb0 tx=0x7fa620009b30).stop 2024-01-13T11:57:02.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.993+0000 7fa635f1e700 1 -- 172.21.15.123:0/94480548 shutdown_connections 2024-01-13T11:57:02.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.993+0000 7fa635f1e700 1 --2- 172.21.15.123:0/94480548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6300601c0 0x7fa6300f91b0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:02.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.993+0000 7fa635f1e700 1 -- 172.21.15.123:0/94480548 >> 172.21.15.123:0/94480548 conn(0x7fa6300f0b70 msgr2=0x7fa6300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:02.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.993+0000 7fa635f1e700 1 -- 172.21.15.123:0/94480548 shutdown_connections 2024-01-13T11:57:02.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.993+0000 7fa635f1e700 1 -- 172.21.15.123:0/94480548 wait complete. 2024-01-13T11:57:02.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.993+0000 7fa635f1e700 1 Processor -- start 2024-01-13T11:57:02.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.993+0000 7fa635f1e700 1 -- start start 2024-01-13T11:57:02.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.993+0000 7fa635f1e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6300601c0 0x7fa6301066f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:02.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.993+0000 7fa635f1e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa630106be0 con 0x7fa6300601c0 2024-01-13T11:57:02.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.994+0000 7fa62f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6300601c0 0x7fa6301066f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:02.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.994+0000 7fa62f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6300601c0 0x7fa6301066f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:44548/0 (socket says 172.21.15.123:44548) 2024-01-13T11:57:02.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.994+0000 7fa62f7fe700 1 -- 172.21.15.123:0/2961533216 learned_addr learned my addr 172.21.15.123:0/2961533216 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:02.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.994+0000 7fa62f7fe700 1 -- 172.21.15.123:0/2961533216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa620005040 con 0x7fa6300601c0 2024-01-13T11:57:02.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.994+0000 7fa62f7fe700 1 --2- 172.21.15.123:0/2961533216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6300601c0 0x7fa6301066f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fa620013510 tx=0x7fa62000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:02.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.994+0000 7fa617fff700 1 -- 172.21.15.123:0/2961533216 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa620009320 con 0x7fa6300601c0 2024-01-13T11:57:02.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.994+0000 7fa635f1e700 1 -- 172.21.15.123:0/2961533216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa630106de0 con 0x7fa6300601c0 2024-01-13T11:57:02.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.994+0000 7fa617fff700 1 -- 172.21.15.123:0/2961533216 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa620013f00 con 0x7fa6300601c0 2024-01-13T11:57:02.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.994+0000 7fa617fff700 1 -- 172.21.15.123:0/2961533216 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa620014dc0 con 0x7fa6300601c0 2024-01-13T11:57:02.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.994+0000 7fa635f1e700 1 -- 172.21.15.123:0/2961533216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa630107270 con 0x7fa6300601c0 2024-01-13T11:57:02.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.995+0000 7fa617fff700 1 -- 172.21.15.123:0/2961533216 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6200095f0 con 0x7fa6300601c0 2024-01-13T11:57:02.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.995+0000 7fa635f1e700 1 -- 172.21.15.123:0/2961533216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa630045ad0 con 0x7fa6300601c0 2024-01-13T11:57:02.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.995+0000 7fa617fff700 1 --2- 172.21.15.123:0/2961533216 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa61802ffb0 0x7fa618032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:02.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.995+0000 7fa617fff700 1 -- 172.21.15.123:0/2961533216 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa620040ff0 con 0x7fa6300601c0 2024-01-13T11:57:02.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.995+0000 7fa62effd700 1 --2- 172.21.15.123:0/2961533216 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa61802ffb0 0x7fa618032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:02.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:02.996+0000 7fa62effd700 1 --2- 172.21.15.123:0/2961533216 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa61802ffb0 0x7fa618032470 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fa6240096a0 tx=0x7fa624006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:02.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:03.000+0000 7fa617fff700 1 -- 172.21.15.123:0/2961533216 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa62001c320 con 0x7fa6300601c0 2024-01-13T11:57:03.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:03.205+0000 7fa635f1e700 1 -- 172.21.15.123:0/2961533216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa630108120 con 0x7fa6300601c0 2024-01-13T11:57:03.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:03.205+0000 7fa617fff700 1 -- 172.21.15.123:0/2961533216 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa62000f340 con 0x7fa6300601c0 2024-01-13T11:57:03.194 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:03.194 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:03.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:03.207+0000 7fa635f1e700 1 -- 172.21.15.123:0/2961533216 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa61802ffb0 msgr2=0x7fa618032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:03.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:03.207+0000 7fa635f1e700 1 --2- 172.21.15.123:0/2961533216 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa61802ffb0 0x7fa618032470 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fa6240096a0 tx=0x7fa624006b40).stop 2024-01-13T11:57:03.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:03.207+0000 7fa635f1e700 1 -- 172.21.15.123:0/2961533216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6300601c0 msgr2=0x7fa6301066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:03.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:03.207+0000 7fa635f1e700 1 --2- 172.21.15.123:0/2961533216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6300601c0 0x7fa6301066f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fa620013510 tx=0x7fa62000b040).stop 2024-01-13T11:57:03.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:03.207+0000 7fa635f1e700 1 -- 172.21.15.123:0/2961533216 shutdown_connections 2024-01-13T11:57:03.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:03.207+0000 7fa635f1e700 1 --2- 172.21.15.123:0/2961533216 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa61802ffb0 0x7fa618032470 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:03.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:03.207+0000 7fa635f1e700 1 --2- 172.21.15.123:0/2961533216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6300601c0 0x7fa6301066f0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:03.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:03.207+0000 7fa635f1e700 1 -- 172.21.15.123:0/2961533216 >> 172.21.15.123:0/2961533216 conn(0x7fa6300f0b70 msgr2=0x7fa6300f1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:03.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:03.207+0000 7fa635f1e700 1 -- 172.21.15.123:0/2961533216 shutdown_connections 2024-01-13T11:57:03.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:03.207+0000 7fa635f1e700 1 -- 172.21.15.123:0/2961533216 wait complete. 2024-01-13T11:57:03.197 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:03.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:03 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:01.570891+0000 mgr.smithi059.hmyldj (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:03.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:03 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:03.206877+0000 mon.smithi059 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.123:0/2961533216' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:04.606 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:04.606 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:05.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:05 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:03.571198+0000 mgr.smithi059.hmyldj (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:05.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.816+0000 7ff653839700 1 -- 172.21.15.123:0/769536246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff64c0f6510 msgr2=0x7ff64c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:05.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.816+0000 7ff653839700 1 --2- 172.21.15.123:0/769536246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff64c0f6510 0x7ff64c0f6930 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7ff640009fb0 tx=0x7ff640009b30).stop 2024-01-13T11:57:05.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.816+0000 7ff653839700 1 -- 172.21.15.123:0/769536246 shutdown_connections 2024-01-13T11:57:05.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.816+0000 7ff653839700 1 --2- 172.21.15.123:0/769536246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff64c0f6510 0x7ff64c0f6930 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:05.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.816+0000 7ff653839700 1 -- 172.21.15.123:0/769536246 >> 172.21.15.123:0/769536246 conn(0x7ff64c0f1c70 msgr2=0x7ff64c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:05.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.816+0000 7ff653839700 1 -- 172.21.15.123:0/769536246 shutdown_connections 2024-01-13T11:57:05.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.816+0000 7ff653839700 1 -- 172.21.15.123:0/769536246 wait complete. 2024-01-13T11:57:05.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.817+0000 7ff653839700 1 Processor -- start 2024-01-13T11:57:05.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.817+0000 7ff653839700 1 -- start start 2024-01-13T11:57:05.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.817+0000 7ff653839700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff64c0f6510 0x7ff64c108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:05.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.817+0000 7ff653839700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c108e00 con 0x7ff64c0f6510 2024-01-13T11:57:05.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.817+0000 7ff6515d5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff64c0f6510 0x7ff64c108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:05.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.817+0000 7ff6515d5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff64c0f6510 0x7ff64c108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:44574/0 (socket says 172.21.15.123:44574) 2024-01-13T11:57:05.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.817+0000 7ff6515d5700 1 -- 172.21.15.123:0/738693636 learned_addr learned my addr 172.21.15.123:0/738693636 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:05.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.818+0000 7ff6515d5700 1 -- 172.21.15.123:0/738693636 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff640005040 con 0x7ff64c0f6510 2024-01-13T11:57:05.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.818+0000 7ff6515d5700 1 --2- 172.21.15.123:0/738693636 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff64c0f6510 0x7ff64c108910 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7ff640013510 tx=0x7ff64000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:05.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.818+0000 7ff63dffb700 1 -- 172.21.15.123:0/738693636 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff640009320 con 0x7ff64c0f6510 2024-01-13T11:57:05.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.818+0000 7ff653839700 1 -- 172.21.15.123:0/738693636 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64c109000 con 0x7ff64c0f6510 2024-01-13T11:57:05.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.818+0000 7ff63dffb700 1 -- 172.21.15.123:0/738693636 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff640013f00 con 0x7ff64c0f6510 2024-01-13T11:57:05.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.818+0000 7ff63dffb700 1 -- 172.21.15.123:0/738693636 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff640014d90 con 0x7ff64c0f6510 2024-01-13T11:57:05.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.818+0000 7ff653839700 1 -- 172.21.15.123:0/738693636 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64c109490 con 0x7ff64c0f6510 2024-01-13T11:57:05.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.819+0000 7ff63dffb700 1 -- 172.21.15.123:0/738693636 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff6400263a0 con 0x7ff64c0f6510 2024-01-13T11:57:05.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.819+0000 7ff653839700 1 -- 172.21.15.123:0/738693636 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff64c045ad0 con 0x7ff64c0f6510 2024-01-13T11:57:05.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.819+0000 7ff63dffb700 1 --2- 172.21.15.123:0/738693636 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff63802ff60 0x7ff638032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:05.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.819+0000 7ff63dffb700 1 -- 172.21.15.123:0/738693636 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff6400412d0 con 0x7ff64c0f6510 2024-01-13T11:57:05.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.819+0000 7ff650dd4700 1 --2- 172.21.15.123:0/738693636 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff63802ff60 0x7ff638032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:05.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.820+0000 7ff650dd4700 1 --2- 172.21.15.123:0/738693636 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff63802ff60 0x7ff638032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7ff6480096a0 tx=0x7ff648006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:05.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:05.824+0000 7ff63dffb700 1 -- 172.21.15.123:0/738693636 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff6400126f0 con 0x7ff64c0f6510 2024-01-13T11:57:06.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:06.028+0000 7ff653839700 1 -- 172.21.15.123:0/738693636 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff64c10a2e0 con 0x7ff64c0f6510 2024-01-13T11:57:06.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:06.028+0000 7ff63dffb700 1 -- 172.21.15.123:0/738693636 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff64000f340 con 0x7ff64c0f6510 2024-01-13T11:57:06.016 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:06.016 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:06.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:06.030+0000 7ff653839700 1 -- 172.21.15.123:0/738693636 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff63802ff60 msgr2=0x7ff638032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:06.030+0000 7ff653839700 1 --2- 172.21.15.123:0/738693636 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff63802ff60 0x7ff638032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7ff6480096a0 tx=0x7ff648006b40).stop 2024-01-13T11:57:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:06.030+0000 7ff653839700 1 -- 172.21.15.123:0/738693636 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff64c0f6510 msgr2=0x7ff64c108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:06.030+0000 7ff653839700 1 --2- 172.21.15.123:0/738693636 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff64c0f6510 0x7ff64c108910 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7ff640013510 tx=0x7ff64000b040).stop 2024-01-13T11:57:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:06.030+0000 7ff653839700 1 -- 172.21.15.123:0/738693636 shutdown_connections 2024-01-13T11:57:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:06.030+0000 7ff653839700 1 --2- 172.21.15.123:0/738693636 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff63802ff60 0x7ff638032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:06.030+0000 7ff653839700 1 --2- 172.21.15.123:0/738693636 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff64c0f6510 0x7ff64c108910 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:06.030+0000 7ff653839700 1 -- 172.21.15.123:0/738693636 >> 172.21.15.123:0/738693636 conn(0x7ff64c0f1c70 msgr2=0x7ff64c0f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:06.030+0000 7ff653839700 1 -- 172.21.15.123:0/738693636 shutdown_connections 2024-01-13T11:57:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:06.030+0000 7ff653839700 1 -- 172.21.15.123:0/738693636 wait complete. 2024-01-13T11:57:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:06.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:06 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:06.029698+0000 mon.smithi059 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.123:0/738693636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:07.403 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:07.403 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:07.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:05.571554+0000 mgr.smithi059.hmyldj (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:08.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.565+0000 7f3fbda6d700 1 -- 172.21.15.123:0/1589698233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb80f5410 msgr2=0x7f3fb80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:08.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.565+0000 7f3fbda6d700 1 --2- 172.21.15.123:0/1589698233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb80f5410 0x7f3fb80f5830 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f3fa8009fb0 tx=0x7f3fa8009b30).stop 2024-01-13T11:57:08.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.565+0000 7f3fbda6d700 1 -- 172.21.15.123:0/1589698233 shutdown_connections 2024-01-13T11:57:08.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.565+0000 7f3fbda6d700 1 --2- 172.21.15.123:0/1589698233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb80f5410 0x7f3fb80f5830 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:08.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.565+0000 7f3fbda6d700 1 -- 172.21.15.123:0/1589698233 >> 172.21.15.123:0/1589698233 conn(0x7f3fb80f0b70 msgr2=0x7f3fb80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:08.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.565+0000 7f3fbda6d700 1 -- 172.21.15.123:0/1589698233 shutdown_connections 2024-01-13T11:57:08.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.565+0000 7f3fbda6d700 1 -- 172.21.15.123:0/1589698233 wait complete. 2024-01-13T11:57:08.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.565+0000 7f3fbda6d700 1 Processor -- start 2024-01-13T11:57:08.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.566+0000 7f3fbda6d700 1 -- start start 2024-01-13T11:57:08.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.566+0000 7f3fbda6d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb80f5410 0x7f3fb8106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:08.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.566+0000 7f3fbda6d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb8106c10 con 0x7f3fb80f5410 2024-01-13T11:57:08.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.566+0000 7f3fb77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb80f5410 0x7f3fb8106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:08.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.566+0000 7f3fb77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb80f5410 0x7f3fb8106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:60572/0 (socket says 172.21.15.123:60572) 2024-01-13T11:57:08.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.566+0000 7f3fb77fe700 1 -- 172.21.15.123:0/2158667897 learned_addr learned my addr 172.21.15.123:0/2158667897 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:08.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.566+0000 7f3fb77fe700 1 -- 172.21.15.123:0/2158667897 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa8005040 con 0x7f3fb80f5410 2024-01-13T11:57:08.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.566+0000 7f3fb77fe700 1 --2- 172.21.15.123:0/2158667897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb80f5410 0x7f3fb8106720 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f3fa8013510 tx=0x7f3fa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:08.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.567+0000 7f3faffff700 1 -- 172.21.15.123:0/2158667897 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fa8009320 con 0x7f3fb80f5410 2024-01-13T11:57:08.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.567+0000 7f3faffff700 1 -- 172.21.15.123:0/2158667897 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fa8013f00 con 0x7f3fb80f5410 2024-01-13T11:57:08.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.567+0000 7f3fbda6d700 1 -- 172.21.15.123:0/2158667897 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb8106e10 con 0x7f3fb80f5410 2024-01-13T11:57:08.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.567+0000 7f3faffff700 1 -- 172.21.15.123:0/2158667897 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fa8014d90 con 0x7f3fb80f5410 2024-01-13T11:57:08.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.567+0000 7f3fbda6d700 1 -- 172.21.15.123:0/2158667897 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fb81072a0 con 0x7f3fb80f5410 2024-01-13T11:57:08.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.567+0000 7f3faffff700 1 -- 172.21.15.123:0/2158667897 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3fa80095f0 con 0x7f3fb80f5410 2024-01-13T11:57:08.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.568+0000 7f3fbda6d700 1 -- 172.21.15.123:0/2158667897 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fb8045ad0 con 0x7f3fb80f5410 2024-01-13T11:57:08.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.568+0000 7f3faffff700 1 --2- 172.21.15.123:0/2158667897 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f3fa402ff60 0x7f3fa4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:08.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.568+0000 7f3faffff700 1 -- 172.21.15.123:0/2158667897 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3fa80410c0 con 0x7f3fb80f5410 2024-01-13T11:57:08.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.568+0000 7f3fb6ffd700 1 --2- 172.21.15.123:0/2158667897 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f3fa402ff60 0x7f3fa4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:08.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.568+0000 7f3fb6ffd700 1 --2- 172.21.15.123:0/2158667897 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f3fa402ff60 0x7f3fa4032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f3fa00096a0 tx=0x7f3fa0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:08.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.572+0000 7f3faffff700 1 -- 172.21.15.123:0/2158667897 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3fa8014420 con 0x7f3fb80f5410 2024-01-13T11:57:08.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.779+0000 7f3fbda6d700 1 -- 172.21.15.123:0/2158667897 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3fb8108150 con 0x7f3fb80f5410 2024-01-13T11:57:08.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.780+0000 7f3faffff700 1 -- 172.21.15.123:0/2158667897 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3fa8010230 con 0x7f3fb80f5410 2024-01-13T11:57:08.767 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:08.768 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:08.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.781+0000 7f3fbda6d700 1 -- 172.21.15.123:0/2158667897 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f3fa402ff60 msgr2=0x7f3fa4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:08.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.781+0000 7f3fbda6d700 1 --2- 172.21.15.123:0/2158667897 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f3fa402ff60 0x7f3fa4032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f3fa00096a0 tx=0x7f3fa0006b40).stop 2024-01-13T11:57:08.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.781+0000 7f3fbda6d700 1 -- 172.21.15.123:0/2158667897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb80f5410 msgr2=0x7f3fb8106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:08.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.781+0000 7f3fbda6d700 1 --2- 172.21.15.123:0/2158667897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb80f5410 0x7f3fb8106720 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f3fa8013510 tx=0x7f3fa800b040).stop 2024-01-13T11:57:08.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.781+0000 7f3fbda6d700 1 -- 172.21.15.123:0/2158667897 shutdown_connections 2024-01-13T11:57:08.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.781+0000 7f3fbda6d700 1 --2- 172.21.15.123:0/2158667897 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f3fa402ff60 0x7f3fa4032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:08.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.781+0000 7f3fbda6d700 1 --2- 172.21.15.123:0/2158667897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb80f5410 0x7f3fb8106720 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:08.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.781+0000 7f3fbda6d700 1 -- 172.21.15.123:0/2158667897 >> 172.21.15.123:0/2158667897 conn(0x7f3fb80f0b70 msgr2=0x7f3fb805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:08.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.781+0000 7f3fbda6d700 1 -- 172.21.15.123:0/2158667897 shutdown_connections 2024-01-13T11:57:08.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:08.781+0000 7f3fbda6d700 1 -- 172.21.15.123:0/2158667897 wait complete. 2024-01-13T11:57:08.770 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:09.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:09 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:07.571797+0000 mgr.smithi059.hmyldj (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:09.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:09 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:08.781326+0000 mon.smithi059 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.123:0/2158667897' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:10.146 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:10.147 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:11.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.317+0000 7f249216b700 1 -- 172.21.15.123:0/2167214695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f248c060280 msgr2=0x7f248c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:11.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.317+0000 7f249216b700 1 --2- 172.21.15.123:0/2167214695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f248c060280 0x7f248c0f90f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f2480009fb0 tx=0x7f2480009b30).stop 2024-01-13T11:57:11.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.317+0000 7f249216b700 1 -- 172.21.15.123:0/2167214695 shutdown_connections 2024-01-13T11:57:11.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.317+0000 7f249216b700 1 --2- 172.21.15.123:0/2167214695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f248c060280 0x7f248c0f90f0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:11.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.317+0000 7f249216b700 1 -- 172.21.15.123:0/2167214695 >> 172.21.15.123:0/2167214695 conn(0x7f248c0f0b70 msgr2=0x7f248c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:11.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.317+0000 7f249216b700 1 -- 172.21.15.123:0/2167214695 shutdown_connections 2024-01-13T11:57:11.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.317+0000 7f249216b700 1 -- 172.21.15.123:0/2167214695 wait complete. 2024-01-13T11:57:11.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.317+0000 7f249216b700 1 Processor -- start 2024-01-13T11:57:11.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.318+0000 7f249216b700 1 -- start start 2024-01-13T11:57:11.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.318+0000 7f249216b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f248c060280 0x7f248c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:11.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.318+0000 7f249216b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248c106bd0 con 0x7f248c060280 2024-01-13T11:57:11.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.318+0000 7f248b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f248c060280 0x7f248c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:11.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.318+0000 7f248b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f248c060280 0x7f248c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:60594/0 (socket says 172.21.15.123:60594) 2024-01-13T11:57:11.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.318+0000 7f248b7fe700 1 -- 172.21.15.123:0/1271414282 learned_addr learned my addr 172.21.15.123:0/1271414282 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:11.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.318+0000 7f248b7fe700 1 -- 172.21.15.123:0/1271414282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2480005040 con 0x7f248c060280 2024-01-13T11:57:11.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.318+0000 7f248b7fe700 1 --2- 172.21.15.123:0/1271414282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f248c060280 0x7f248c1066e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f2480013510 tx=0x7f248000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:11.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.318+0000 7f2473fff700 1 -- 172.21.15.123:0/1271414282 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2480009320 con 0x7f248c060280 2024-01-13T11:57:11.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.318+0000 7f249216b700 1 -- 172.21.15.123:0/1271414282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f248c106dd0 con 0x7f248c060280 2024-01-13T11:57:11.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.318+0000 7f2473fff700 1 -- 172.21.15.123:0/1271414282 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2480013f00 con 0x7f248c060280 2024-01-13T11:57:11.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.318+0000 7f2473fff700 1 -- 172.21.15.123:0/1271414282 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2480014dc0 con 0x7f248c060280 2024-01-13T11:57:11.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.319+0000 7f249216b700 1 -- 172.21.15.123:0/1271414282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f248c107260 con 0x7f248c060280 2024-01-13T11:57:11.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.319+0000 7f2473fff700 1 -- 172.21.15.123:0/1271414282 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f24800095f0 con 0x7f248c060280 2024-01-13T11:57:11.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.319+0000 7f249216b700 1 -- 172.21.15.123:0/1271414282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f248c045ad0 con 0x7f248c060280 2024-01-13T11:57:11.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.319+0000 7f2473fff700 1 --2- 172.21.15.123:0/1271414282 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f247402ff60 0x7f2474032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:11.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.319+0000 7f2473fff700 1 -- 172.21.15.123:0/1271414282 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f24800410c0 con 0x7f248c060280 2024-01-13T11:57:11.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.320+0000 7f248affd700 1 --2- 172.21.15.123:0/1271414282 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f247402ff60 0x7f2474032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:11.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.320+0000 7f248affd700 1 --2- 172.21.15.123:0/1271414282 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f247402ff60 0x7f2474032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f247c0096a0 tx=0x7f247c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:11.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.324+0000 7f2473fff700 1 -- 172.21.15.123:0/1271414282 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f248001c790 con 0x7f248c060280 2024-01-13T11:57:11.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.529+0000 7f249216b700 1 -- 172.21.15.123:0/1271414282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f248c108110 con 0x7f248c060280 2024-01-13T11:57:11.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.529+0000 7f2473fff700 1 -- 172.21.15.123:0/1271414282 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2480010230 con 0x7f248c060280 2024-01-13T11:57:11.517 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:11.517 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:11.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.531+0000 7f249216b700 1 -- 172.21.15.123:0/1271414282 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f247402ff60 msgr2=0x7f2474032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:11.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.531+0000 7f249216b700 1 --2- 172.21.15.123:0/1271414282 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f247402ff60 0x7f2474032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f247c0096a0 tx=0x7f247c006b40).stop 2024-01-13T11:57:11.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.531+0000 7f249216b700 1 -- 172.21.15.123:0/1271414282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f248c060280 msgr2=0x7f248c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:11.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.531+0000 7f249216b700 1 --2- 172.21.15.123:0/1271414282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f248c060280 0x7f248c1066e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f2480013510 tx=0x7f248000b040).stop 2024-01-13T11:57:11.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.531+0000 7f249216b700 1 -- 172.21.15.123:0/1271414282 shutdown_connections 2024-01-13T11:57:11.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.531+0000 7f249216b700 1 --2- 172.21.15.123:0/1271414282 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f247402ff60 0x7f2474032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:11.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.531+0000 7f249216b700 1 --2- 172.21.15.123:0/1271414282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f248c060280 0x7f248c1066e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:11.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.531+0000 7f249216b700 1 -- 172.21.15.123:0/1271414282 >> 172.21.15.123:0/1271414282 conn(0x7f248c0f0b70 msgr2=0x7f248c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:11.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.531+0000 7f249216b700 1 -- 172.21.15.123:0/1271414282 shutdown_connections 2024-01-13T11:57:11.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:11.531+0000 7f249216b700 1 -- 172.21.15.123:0/1271414282 wait complete. 2024-01-13T11:57:11.520 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:11.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:11 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:09.572088+0000 mgr.smithi059.hmyldj (mgr.14162) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:12.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:12 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:11.530751+0000 mon.smithi059 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.123:0/1271414282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:12.941 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:12.942 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:13.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:13 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:11.572369+0000 mgr.smithi059.hmyldj (mgr.14162) 232 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:14.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.186+0000 7f135b7bc700 1 -- 172.21.15.123:0/3762161860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13540f5410 msgr2=0x7f13540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:14.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.186+0000 7f135b7bc700 1 --2- 172.21.15.123:0/3762161860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13540f5410 0x7f13540f5830 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f1348009fb0 tx=0x7f1348009b30).stop 2024-01-13T11:57:14.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.186+0000 7f135b7bc700 1 -- 172.21.15.123:0/3762161860 shutdown_connections 2024-01-13T11:57:14.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.186+0000 7f135b7bc700 1 --2- 172.21.15.123:0/3762161860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13540f5410 0x7f13540f5830 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:14.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.186+0000 7f135b7bc700 1 -- 172.21.15.123:0/3762161860 >> 172.21.15.123:0/3762161860 conn(0x7f13540f0b70 msgr2=0x7f13540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:14.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.186+0000 7f135b7bc700 1 -- 172.21.15.123:0/3762161860 shutdown_connections 2024-01-13T11:57:14.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.186+0000 7f135b7bc700 1 -- 172.21.15.123:0/3762161860 wait complete. 2024-01-13T11:57:14.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.187+0000 7f135b7bc700 1 Processor -- start 2024-01-13T11:57:14.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.187+0000 7f135b7bc700 1 -- start start 2024-01-13T11:57:14.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.187+0000 7f135b7bc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13540f5410 0x7f13540fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:14.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.187+0000 7f135b7bc700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13540fb820 con 0x7f13540f5410 2024-01-13T11:57:14.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.187+0000 7f1359558700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13540f5410 0x7f13540fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:14.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.187+0000 7f1359558700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13540f5410 0x7f13540fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:60618/0 (socket says 172.21.15.123:60618) 2024-01-13T11:57:14.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.187+0000 7f1359558700 1 -- 172.21.15.123:0/709363055 learned_addr learned my addr 172.21.15.123:0/709363055 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:14.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.188+0000 7f1359558700 1 -- 172.21.15.123:0/709363055 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1348005040 con 0x7f13540f5410 2024-01-13T11:57:14.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.188+0000 7f1359558700 1 --2- 172.21.15.123:0/709363055 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13540f5410 0x7f13540fd3a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f1348013510 tx=0x7f134800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:14.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.188+0000 7f1345ffb700 1 -- 172.21.15.123:0/709363055 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1348009320 con 0x7f13540f5410 2024-01-13T11:57:14.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.188+0000 7f135b7bc700 1 -- 172.21.15.123:0/709363055 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13540fb9c0 con 0x7f13540f5410 2024-01-13T11:57:14.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.188+0000 7f1345ffb700 1 -- 172.21.15.123:0/709363055 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1348013f00 con 0x7f13540f5410 2024-01-13T11:57:14.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.188+0000 7f1345ffb700 1 -- 172.21.15.123:0/709363055 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1348014d90 con 0x7f13540f5410 2024-01-13T11:57:14.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.188+0000 7f135b7bc700 1 -- 172.21.15.123:0/709363055 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13540fbe50 con 0x7f13540f5410 2024-01-13T11:57:14.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.189+0000 7f1345ffb700 1 -- 172.21.15.123:0/709363055 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13480095f0 con 0x7f13540f5410 2024-01-13T11:57:14.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.189+0000 7f135b7bc700 1 -- 172.21.15.123:0/709363055 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1354059100 con 0x7f13540f5410 2024-01-13T11:57:14.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.189+0000 7f1345ffb700 1 --2- 172.21.15.123:0/709363055 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f134002ff60 0x7f1340032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:14.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.189+0000 7f1345ffb700 1 -- 172.21.15.123:0/709363055 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1348041000 con 0x7f13540f5410 2024-01-13T11:57:14.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.189+0000 7f1358d57700 1 --2- 172.21.15.123:0/709363055 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f134002ff60 0x7f1340032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:14.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.190+0000 7f1358d57700 1 --2- 172.21.15.123:0/709363055 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f134002ff60 0x7f1340032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f13500096a0 tx=0x7f1350006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:14.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.194+0000 7f1345ffb700 1 -- 172.21.15.123:0/709363055 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1348014420 con 0x7f13540f5410 2024-01-13T11:57:14.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.399+0000 7f135b7bc700 1 -- 172.21.15.123:0/709363055 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13540fae60 con 0x7f13540f5410 2024-01-13T11:57:14.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.400+0000 7f1345ffb700 1 -- 172.21.15.123:0/709363055 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f134801d7f0 con 0x7f13540f5410 2024-01-13T11:57:14.388 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:14.388 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:14.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.402+0000 7f135b7bc700 1 -- 172.21.15.123:0/709363055 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f134002ff60 msgr2=0x7f1340032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:14.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.402+0000 7f135b7bc700 1 --2- 172.21.15.123:0/709363055 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f134002ff60 0x7f1340032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f13500096a0 tx=0x7f1350006b40).stop 2024-01-13T11:57:14.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.402+0000 7f135b7bc700 1 -- 172.21.15.123:0/709363055 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13540f5410 msgr2=0x7f13540fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:14.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.402+0000 7f135b7bc700 1 --2- 172.21.15.123:0/709363055 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13540f5410 0x7f13540fd3a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f1348013510 tx=0x7f134800b040).stop 2024-01-13T11:57:14.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.402+0000 7f135b7bc700 1 -- 172.21.15.123:0/709363055 shutdown_connections 2024-01-13T11:57:14.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.402+0000 7f135b7bc700 1 --2- 172.21.15.123:0/709363055 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f134002ff60 0x7f1340032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:14.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.402+0000 7f135b7bc700 1 --2- 172.21.15.123:0/709363055 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13540f5410 0x7f13540fd3a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:14.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.402+0000 7f135b7bc700 1 -- 172.21.15.123:0/709363055 >> 172.21.15.123:0/709363055 conn(0x7f13540f0b70 msgr2=0x7f135405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:14.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.402+0000 7f135b7bc700 1 -- 172.21.15.123:0/709363055 shutdown_connections 2024-01-13T11:57:14.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:14.402+0000 7f135b7bc700 1 -- 172.21.15.123:0/709363055 wait complete. 2024-01-13T11:57:14.390 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:14.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cephadm 2024-01-13T11:57:13.190459+0000 mgr.smithi059.hmyldj (mgr.14162) 233 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi059 ... 2024-01-13T11:57:14.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Verifying port 9100 ... 2024-01-13T11:57:14.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Non-zero exit code 1 from systemctl start ceph-a3276948-b209-11ee-95ac-87774f69a715@node-exporter.smithi059 2024-01-13T11:57:14.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: systemctl: stderr Job for ceph-a3276948-b209-11ee-95ac-87774f69a715@node-exporter.smithi059.service failed because a timeout was exceeded. 2024-01-13T11:57:14.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: systemctl: stderr See "systemctl status ceph-a3276948-b209-11ee-95ac-87774f69a715@node-exporter.smithi059.service" and "journalctl -xe" for details. 2024-01-13T11:57:14.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Traceback (most recent call last): 2024-01-13T11:57:14.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-13T11:57:14.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: main() 2024-01-13T11:57:14.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-13T11:57:14.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: r = ctx.func(ctx) 2024-01-13T11:57:14.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-13T11:57:14.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: return func(ctx) 2024-01-13T11:57:14.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-13T11:57:14.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: ports=daemon_ports) 2024-01-13T11:57:14.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-13T11:57:14.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-13T11:57:14.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-13T11:57:14.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-13T11:57:14.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-13T11:57:14.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:57:14.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: RuntimeError: Failed command: systemctl start ceph-a3276948-b209-11ee-95ac-87774f69a715@node-exporter.smithi059 2024-01-13T11:57:14.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Traceback (most recent call last): 2024-01-13T11:57:14.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-13T11:57:14.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: yield (conn, connr) 2024-01-13T11:57:14.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-13T11:57:14.638 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: code, '\n'.join(err))) 2024-01-13T11:57:14.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi059 ... 2024-01-13T11:57:14.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Verifying port 9100 ... 2024-01-13T11:57:14.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Non-zero exit code 1 from systemctl start ceph-a3276948-b209-11ee-95ac-87774f69a715@node-exporter.smithi059 2024-01-13T11:57:14.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: systemctl: stderr Job for ceph-a3276948-b209-11ee-95ac-87774f69a715@node-exporter.smithi059.service failed because a timeout was exceeded. 2024-01-13T11:57:14.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: systemctl: stderr See "systemctl status ceph-a3276948-b209-11ee-95ac-87774f69a715@node-exporter.smithi059.service" and "journalctl -xe" for details. 2024-01-13T11:57:14.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: Traceback (most recent call last): 2024-01-13T11:57:14.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-13T11:57:14.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: main() 2024-01-13T11:57:14.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-13T11:57:14.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: r = ctx.func(ctx) 2024-01-13T11:57:14.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-13T11:57:14.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: return func(ctx) 2024-01-13T11:57:14.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-13T11:57:14.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: ports=daemon_ports) 2024-01-13T11:57:14.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-13T11:57:14.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-13T11:57:14.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-13T11:57:14.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-13T11:57:14.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: File "/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-13T11:57:14.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:57:14.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: RuntimeError: Failed command: systemctl start ceph-a3276948-b209-11ee-95ac-87774f69a715@node-exporter.smithi059 2024-01-13T11:57:14.640 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:14 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cephadm 2024-01-13T11:57:13.196632+0000 mgr.smithi059.hmyldj (mgr.14162) 234 : cephadm [INF] Deploying daemon prometheus.smithi059 on smithi059 2024-01-13T11:57:15.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:15 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:13.572641+0000 mgr.smithi059.hmyldj (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:15.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:15 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:14.401098+0000 mon.smithi059 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.123:0/709363055' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:15.786 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:15.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:16.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.973+0000 7fd49e2b2700 1 -- 172.21.15.123:0/2244944353 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4980f5410 msgr2=0x7fd4980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:16.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.973+0000 7fd49e2b2700 1 --2- 172.21.15.123:0/2244944353 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4980f5410 0x7fd4980f5830 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fd488009fb0 tx=0x7fd488009b30).stop 2024-01-13T11:57:16.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.973+0000 7fd49e2b2700 1 -- 172.21.15.123:0/2244944353 shutdown_connections 2024-01-13T11:57:16.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.973+0000 7fd49e2b2700 1 --2- 172.21.15.123:0/2244944353 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4980f5410 0x7fd4980f5830 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:16.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.973+0000 7fd49e2b2700 1 -- 172.21.15.123:0/2244944353 >> 172.21.15.123:0/2244944353 conn(0x7fd4980f0b70 msgr2=0x7fd4980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:16.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.973+0000 7fd49e2b2700 1 -- 172.21.15.123:0/2244944353 shutdown_connections 2024-01-13T11:57:16.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.973+0000 7fd49e2b2700 1 -- 172.21.15.123:0/2244944353 wait complete. 2024-01-13T11:57:16.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.974+0000 7fd49e2b2700 1 Processor -- start 2024-01-13T11:57:16.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.974+0000 7fd49e2b2700 1 -- start start 2024-01-13T11:57:16.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.974+0000 7fd49e2b2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4980f5410 0x7fd498105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:16.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.974+0000 7fd49e2b2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd498105b20 con 0x7fd4980f5410 2024-01-13T11:57:16.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.975+0000 7fd497fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4980f5410 0x7fd498105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:16.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.975+0000 7fd497fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4980f5410 0x7fd498105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:55466/0 (socket says 172.21.15.123:55466) 2024-01-13T11:57:16.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.975+0000 7fd497fff700 1 -- 172.21.15.123:0/3040135990 learned_addr learned my addr 172.21.15.123:0/3040135990 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:16.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.975+0000 7fd497fff700 1 -- 172.21.15.123:0/3040135990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd488005040 con 0x7fd4980f5410 2024-01-13T11:57:16.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.975+0000 7fd497fff700 1 --2- 172.21.15.123:0/3040135990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4980f5410 0x7fd498105630 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fd488013510 tx=0x7fd48800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:16.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.975+0000 7fd494ff9700 1 -- 172.21.15.123:0/3040135990 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd488009320 con 0x7fd4980f5410 2024-01-13T11:57:16.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.975+0000 7fd494ff9700 1 -- 172.21.15.123:0/3040135990 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd488013f00 con 0x7fd4980f5410 2024-01-13T11:57:16.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.975+0000 7fd49e2b2700 1 -- 172.21.15.123:0/3040135990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd498105d20 con 0x7fd4980f5410 2024-01-13T11:57:16.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.975+0000 7fd494ff9700 1 -- 172.21.15.123:0/3040135990 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd488014d90 con 0x7fd4980f5410 2024-01-13T11:57:16.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.975+0000 7fd49e2b2700 1 -- 172.21.15.123:0/3040135990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4981061b0 con 0x7fd4980f5410 2024-01-13T11:57:16.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.976+0000 7fd494ff9700 1 -- 172.21.15.123:0/3040135990 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd4880263a0 con 0x7fd4980f5410 2024-01-13T11:57:16.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.976+0000 7fd494ff9700 1 --2- 172.21.15.123:0/3040135990 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd48002ff60 0x7fd480032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:16.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.976+0000 7fd49e2b2700 1 -- 172.21.15.123:0/3040135990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd498045ad0 con 0x7fd4980f5410 2024-01-13T11:57:16.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.976+0000 7fd494ff9700 1 -- 172.21.15.123:0/3040135990 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd4880410a0 con 0x7fd4980f5410 2024-01-13T11:57:16.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.977+0000 7fd4977fe700 1 --2- 172.21.15.123:0/3040135990 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd48002ff60 0x7fd480032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:16.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.977+0000 7fd4977fe700 1 --2- 172.21.15.123:0/3040135990 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd48002ff60 0x7fd480032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fd48c0096a0 tx=0x7fd48c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:16.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:16.981+0000 7fd494ff9700 1 -- 172.21.15.123:0/3040135990 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd488012920 con 0x7fd4980f5410 2024-01-13T11:57:17.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:17.186+0000 7fd49e2b2700 1 -- 172.21.15.123:0/3040135990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4980faf50 con 0x7fd4980f5410 2024-01-13T11:57:17.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:17.187+0000 7fd494ff9700 1 -- 172.21.15.123:0/3040135990 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd48800f340 con 0x7fd4980f5410 2024-01-13T11:57:17.175 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:17.175 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:17.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:17.189+0000 7fd49e2b2700 1 -- 172.21.15.123:0/3040135990 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd48002ff60 msgr2=0x7fd480032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:17.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:17.189+0000 7fd49e2b2700 1 --2- 172.21.15.123:0/3040135990 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd48002ff60 0x7fd480032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fd48c0096a0 tx=0x7fd48c006b40).stop 2024-01-13T11:57:17.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:17.189+0000 7fd49e2b2700 1 -- 172.21.15.123:0/3040135990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4980f5410 msgr2=0x7fd498105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:17.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:17.189+0000 7fd49e2b2700 1 --2- 172.21.15.123:0/3040135990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4980f5410 0x7fd498105630 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fd488013510 tx=0x7fd48800b040).stop 2024-01-13T11:57:17.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:17.189+0000 7fd49e2b2700 1 -- 172.21.15.123:0/3040135990 shutdown_connections 2024-01-13T11:57:17.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:17.189+0000 7fd49e2b2700 1 --2- 172.21.15.123:0/3040135990 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd48002ff60 0x7fd480032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:17.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:17.189+0000 7fd49e2b2700 1 --2- 172.21.15.123:0/3040135990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4980f5410 0x7fd498105630 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:17.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:17.189+0000 7fd49e2b2700 1 -- 172.21.15.123:0/3040135990 >> 172.21.15.123:0/3040135990 conn(0x7fd4980f0b70 msgr2=0x7fd4980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:17.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:17.189+0000 7fd49e2b2700 1 -- 172.21.15.123:0/3040135990 shutdown_connections 2024-01-13T11:57:17.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:17.189+0000 7fd49e2b2700 1 -- 172.21.15.123:0/3040135990 wait complete. 2024-01-13T11:57:17.178 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:17.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:17 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:15.573087+0000 mgr.smithi059.hmyldj (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:17.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:17 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:17.188613+0000 mon.smithi059 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.123:0/3040135990' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:18.555 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:18.555 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:19.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:19 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:17.573417+0000 mgr.smithi059.hmyldj (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:19.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.742+0000 7f8f6f3c0700 1 -- 172.21.15.123:0/610892732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f680f6510 msgr2=0x7f8f680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:19.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.742+0000 7f8f6f3c0700 1 --2- 172.21.15.123:0/610892732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f680f6510 0x7f8f680f6930 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f8f58009fb0 tx=0x7f8f58009b30).stop 2024-01-13T11:57:19.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.742+0000 7f8f6f3c0700 1 -- 172.21.15.123:0/610892732 shutdown_connections 2024-01-13T11:57:19.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.742+0000 7f8f6f3c0700 1 --2- 172.21.15.123:0/610892732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f680f6510 0x7f8f680f6930 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:19.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.742+0000 7f8f6f3c0700 1 -- 172.21.15.123:0/610892732 >> 172.21.15.123:0/610892732 conn(0x7f8f680f1c70 msgr2=0x7f8f680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:19.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.742+0000 7f8f6f3c0700 1 -- 172.21.15.123:0/610892732 shutdown_connections 2024-01-13T11:57:19.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.742+0000 7f8f6f3c0700 1 -- 172.21.15.123:0/610892732 wait complete. 2024-01-13T11:57:19.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.743+0000 7f8f6f3c0700 1 Processor -- start 2024-01-13T11:57:19.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.743+0000 7f8f6f3c0700 1 -- start start 2024-01-13T11:57:19.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.743+0000 7f8f6f3c0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f680f6510 0x7f8f6806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:19.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.743+0000 7f8f6f3c0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f68069e20 con 0x7f8f680f6510 2024-01-13T11:57:19.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.743+0000 7f8f6d15c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f680f6510 0x7f8f6806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:19.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.743+0000 7f8f6d15c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f680f6510 0x7f8f6806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:55488/0 (socket says 172.21.15.123:55488) 2024-01-13T11:57:19.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.743+0000 7f8f6d15c700 1 -- 172.21.15.123:0/119898639 learned_addr learned my addr 172.21.15.123:0/119898639 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:19.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.743+0000 7f8f6d15c700 1 -- 172.21.15.123:0/119898639 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f58005040 con 0x7f8f680f6510 2024-01-13T11:57:19.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.743+0000 7f8f6d15c700 1 --2- 172.21.15.123:0/119898639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f680f6510 0x7f8f6806b9a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f8f58014000 tx=0x7f8f5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:19.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.744+0000 7f8f5dffb700 1 -- 172.21.15.123:0/119898639 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f580095f0 con 0x7f8f680f6510 2024-01-13T11:57:19.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.744+0000 7f8f5dffb700 1 -- 172.21.15.123:0/119898639 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f58007e70 con 0x7f8f680f6510 2024-01-13T11:57:19.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.744+0000 7f8f6f3c0700 1 -- 172.21.15.123:0/119898639 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f68069fc0 con 0x7f8f680f6510 2024-01-13T11:57:19.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.744+0000 7f8f5dffb700 1 -- 172.21.15.123:0/119898639 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f58014ea0 con 0x7f8f680f6510 2024-01-13T11:57:19.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.744+0000 7f8f6f3c0700 1 -- 172.21.15.123:0/119898639 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6806a450 con 0x7f8f680f6510 2024-01-13T11:57:19.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.745+0000 7f8f6f3c0700 1 -- 172.21.15.123:0/119898639 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f68059100 con 0x7f8f680f6510 2024-01-13T11:57:19.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.745+0000 7f8f5dffb700 1 -- 172.21.15.123:0/119898639 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f58025070 con 0x7f8f680f6510 2024-01-13T11:57:19.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.745+0000 7f8f5dffb700 1 --2- 172.21.15.123:0/119898639 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8f5402ff60 0x7f8f54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:19.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.745+0000 7f8f5dffb700 1 -- 172.21.15.123:0/119898639 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f58040970 con 0x7f8f680f6510 2024-01-13T11:57:19.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.745+0000 7f8f6c95b700 1 --2- 172.21.15.123:0/119898639 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8f5402ff60 0x7f8f54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:19.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.745+0000 7f8f6c95b700 1 --2- 172.21.15.123:0/119898639 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8f5402ff60 0x7f8f54032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f8f640096a0 tx=0x7f8f64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:19.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.750+0000 7f8f5dffb700 1 -- 172.21.15.123:0/119898639 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f580079c0 con 0x7f8f680f6510 2024-01-13T11:57:19.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.956+0000 7f8f6f3c0700 1 -- 172.21.15.123:0/119898639 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f680fbf60 con 0x7f8f680f6510 2024-01-13T11:57:19.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.957+0000 7f8f5dffb700 1 -- 172.21.15.123:0/119898639 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8f5800f300 con 0x7f8f680f6510 2024-01-13T11:57:19.944 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:19.945 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:19.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.958+0000 7f8f6f3c0700 1 -- 172.21.15.123:0/119898639 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8f5402ff60 msgr2=0x7f8f54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:19.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.958+0000 7f8f6f3c0700 1 --2- 172.21.15.123:0/119898639 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8f5402ff60 0x7f8f54032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f8f640096a0 tx=0x7f8f64006b40).stop 2024-01-13T11:57:19.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.958+0000 7f8f6f3c0700 1 -- 172.21.15.123:0/119898639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f680f6510 msgr2=0x7f8f6806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:19.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.958+0000 7f8f6f3c0700 1 --2- 172.21.15.123:0/119898639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f680f6510 0x7f8f6806b9a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f8f58014000 tx=0x7f8f5800b040).stop 2024-01-13T11:57:19.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.958+0000 7f8f6f3c0700 1 -- 172.21.15.123:0/119898639 shutdown_connections 2024-01-13T11:57:19.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.958+0000 7f8f6f3c0700 1 --2- 172.21.15.123:0/119898639 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8f5402ff60 0x7f8f54032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:19.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.958+0000 7f8f6f3c0700 1 --2- 172.21.15.123:0/119898639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f680f6510 0x7f8f6806b9a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:19.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.958+0000 7f8f6f3c0700 1 -- 172.21.15.123:0/119898639 >> 172.21.15.123:0/119898639 conn(0x7f8f680f1c70 msgr2=0x7f8f680f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:19.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.958+0000 7f8f6f3c0700 1 -- 172.21.15.123:0/119898639 shutdown_connections 2024-01-13T11:57:19.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:19.958+0000 7f8f6f3c0700 1 -- 172.21.15.123:0/119898639 wait complete. 2024-01-13T11:57:19.947 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:20.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:20 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:19.958310+0000 mon.smithi059 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.123:0/119898639' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:21.351 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:21.352 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:21.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:21 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:19.573808+0000 mgr.smithi059.hmyldj (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:22.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.566+0000 7f41e1216700 1 -- 172.21.15.123:0/709141978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41dc0f5410 msgr2=0x7f41dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:22.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.566+0000 7f41e1216700 1 --2- 172.21.15.123:0/709141978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41dc0f5410 0x7f41dc0f5830 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f41d0009fb0 tx=0x7f41d0009b30).stop 2024-01-13T11:57:22.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.566+0000 7f41e1216700 1 -- 172.21.15.123:0/709141978 shutdown_connections 2024-01-13T11:57:22.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.566+0000 7f41e1216700 1 --2- 172.21.15.123:0/709141978 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41dc0f5410 0x7f41dc0f5830 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:22.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.566+0000 7f41e1216700 1 -- 172.21.15.123:0/709141978 >> 172.21.15.123:0/709141978 conn(0x7f41dc0f0b70 msgr2=0x7f41dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:22.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.566+0000 7f41e1216700 1 -- 172.21.15.123:0/709141978 shutdown_connections 2024-01-13T11:57:22.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.566+0000 7f41e1216700 1 -- 172.21.15.123:0/709141978 wait complete. 2024-01-13T11:57:22.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.567+0000 7f41e1216700 1 Processor -- start 2024-01-13T11:57:22.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.567+0000 7f41e1216700 1 -- start start 2024-01-13T11:57:22.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.567+0000 7f41e1216700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41dc0f5410 0x7f41dc1093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:22.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.567+0000 7f41e1216700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc060280 con 0x7f41dc0f5410 2024-01-13T11:57:22.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.567+0000 7f41dad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41dc0f5410 0x7f41dc1093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:22.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.567+0000 7f41dad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41dc0f5410 0x7f41dc1093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:55518/0 (socket says 172.21.15.123:55518) 2024-01-13T11:57:22.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.567+0000 7f41dad9d700 1 -- 172.21.15.123:0/504833102 learned_addr learned my addr 172.21.15.123:0/504833102 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:22.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.567+0000 7f41dad9d700 1 -- 172.21.15.123:0/504833102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d0005040 con 0x7f41dc0f5410 2024-01-13T11:57:22.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.567+0000 7f41dad9d700 1 --2- 172.21.15.123:0/504833102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41dc0f5410 0x7f41dc1093c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f41d0014000 tx=0x7f41d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:22.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.568+0000 7f41c37fe700 1 -- 172.21.15.123:0/504833102 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41d00095f0 con 0x7f41dc0f5410 2024-01-13T11:57:22.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.568+0000 7f41e1216700 1 -- 172.21.15.123:0/504833102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41dc060480 con 0x7f41dc0f5410 2024-01-13T11:57:22.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.568+0000 7f41c37fe700 1 -- 172.21.15.123:0/504833102 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f41d0007e70 con 0x7f41dc0f5410 2024-01-13T11:57:22.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.568+0000 7f41c37fe700 1 -- 172.21.15.123:0/504833102 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41d0014ea0 con 0x7f41dc0f5410 2024-01-13T11:57:22.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.568+0000 7f41e1216700 1 -- 172.21.15.123:0/504833102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41dc109af0 con 0x7f41dc0f5410 2024-01-13T11:57:22.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.568+0000 7f41c37fe700 1 -- 172.21.15.123:0/504833102 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41d0025070 con 0x7f41dc0f5410 2024-01-13T11:57:22.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.568+0000 7f41e1216700 1 -- 172.21.15.123:0/504833102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41dc045ad0 con 0x7f41dc0f5410 2024-01-13T11:57:22.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.569+0000 7f41c37fe700 1 --2- 172.21.15.123:0/504833102 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f41c402ff60 0x7f41c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:22.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.569+0000 7f41c37fe700 1 -- 172.21.15.123:0/504833102 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f41d00409e0 con 0x7f41dc0f5410 2024-01-13T11:57:22.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.569+0000 7f41da59c700 1 --2- 172.21.15.123:0/504833102 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f41c402ff60 0x7f41c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:22.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.569+0000 7f41da59c700 1 --2- 172.21.15.123:0/504833102 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f41c402ff60 0x7f41c4032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f41cc0096a0 tx=0x7f41cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:22.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.574+0000 7f41c37fe700 1 -- 172.21.15.123:0/504833102 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41d000f4e0 con 0x7f41dc0f5410 2024-01-13T11:57:22.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.782+0000 7f41e1216700 1 -- 172.21.15.123:0/504833102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41dc10a340 con 0x7f41dc0f5410 2024-01-13T11:57:22.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.783+0000 7f41c37fe700 1 -- 172.21.15.123:0/504833102 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f41d000f300 con 0x7f41dc0f5410 2024-01-13T11:57:22.770 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:22.770 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:22.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.784+0000 7f41e1216700 1 -- 172.21.15.123:0/504833102 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f41c402ff60 msgr2=0x7f41c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:22.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.784+0000 7f41e1216700 1 --2- 172.21.15.123:0/504833102 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f41c402ff60 0x7f41c4032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f41cc0096a0 tx=0x7f41cc006b40).stop 2024-01-13T11:57:22.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.784+0000 7f41e1216700 1 -- 172.21.15.123:0/504833102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41dc0f5410 msgr2=0x7f41dc1093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:22.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.784+0000 7f41e1216700 1 --2- 172.21.15.123:0/504833102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41dc0f5410 0x7f41dc1093c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f41d0014000 tx=0x7f41d000b040).stop 2024-01-13T11:57:22.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.784+0000 7f41e1216700 1 -- 172.21.15.123:0/504833102 shutdown_connections 2024-01-13T11:57:22.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.784+0000 7f41e1216700 1 --2- 172.21.15.123:0/504833102 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f41c402ff60 0x7f41c4032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:22.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.784+0000 7f41e1216700 1 --2- 172.21.15.123:0/504833102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41dc0f5410 0x7f41dc1093c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:22.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.784+0000 7f41e1216700 1 -- 172.21.15.123:0/504833102 >> 172.21.15.123:0/504833102 conn(0x7f41dc0f0b70 msgr2=0x7f41dc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:22.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.784+0000 7f41e1216700 1 -- 172.21.15.123:0/504833102 shutdown_connections 2024-01-13T11:57:22.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:22.784+0000 7f41e1216700 1 -- 172.21.15.123:0/504833102 wait complete. 2024-01-13T11:57:22.773 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:23.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:23 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:21.574054+0000 mgr.smithi059.hmyldj (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:23.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:23 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:22.784044+0000 mon.smithi059 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.123:0/504833102' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:24.160 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:24.161 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:25.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.400+0000 7f772e1ed700 1 -- 172.21.15.123:0/828532221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77280f5410 msgr2=0x7f77280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:25.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.400+0000 7f772e1ed700 1 --2- 172.21.15.123:0/828532221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77280f5410 0x7f77280f5830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f7710009fb0 tx=0x7f7710009b30).stop 2024-01-13T11:57:25.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.400+0000 7f772e1ed700 1 -- 172.21.15.123:0/828532221 shutdown_connections 2024-01-13T11:57:25.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.400+0000 7f772e1ed700 1 --2- 172.21.15.123:0/828532221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77280f5410 0x7f77280f5830 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:25.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.400+0000 7f772e1ed700 1 -- 172.21.15.123:0/828532221 >> 172.21.15.123:0/828532221 conn(0x7f77280f0b70 msgr2=0x7f77280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:25.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.400+0000 7f772e1ed700 1 -- 172.21.15.123:0/828532221 shutdown_connections 2024-01-13T11:57:25.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.400+0000 7f772e1ed700 1 -- 172.21.15.123:0/828532221 wait complete. 2024-01-13T11:57:25.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.401+0000 7f772e1ed700 1 Processor -- start 2024-01-13T11:57:25.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.401+0000 7f772e1ed700 1 -- start start 2024-01-13T11:57:25.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.401+0000 7f772e1ed700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77280f5410 0x7f7728109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:25.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.401+0000 7f772e1ed700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7728060220 con 0x7f77280f5410 2024-01-13T11:57:25.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.402+0000 7f77277fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77280f5410 0x7f7728109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:25.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.402+0000 7f77277fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77280f5410 0x7f7728109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:55542/0 (socket says 172.21.15.123:55542) 2024-01-13T11:57:25.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.402+0000 7f77277fe700 1 -- 172.21.15.123:0/2237820028 learned_addr learned my addr 172.21.15.123:0/2237820028 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:25.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.402+0000 7f77277fe700 1 -- 172.21.15.123:0/2237820028 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7710005040 con 0x7f77280f5410 2024-01-13T11:57:25.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.402+0000 7f77277fe700 1 --2- 172.21.15.123:0/2237820028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77280f5410 0x7f7728109450 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f7710013510 tx=0x7f771000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:25.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.402+0000 7f771ffff700 1 -- 172.21.15.123:0/2237820028 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7710009320 con 0x7f77280f5410 2024-01-13T11:57:25.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.402+0000 7f772e1ed700 1 -- 172.21.15.123:0/2237820028 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7728060420 con 0x7f77280f5410 2024-01-13T11:57:25.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.402+0000 7f771ffff700 1 -- 172.21.15.123:0/2237820028 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7710013f00 con 0x7f77280f5410 2024-01-13T11:57:25.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.402+0000 7f771ffff700 1 -- 172.21.15.123:0/2237820028 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7710014d90 con 0x7f77280f5410 2024-01-13T11:57:25.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.402+0000 7f772e1ed700 1 -- 172.21.15.123:0/2237820028 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7728109b00 con 0x7f77280f5410 2024-01-13T11:57:25.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.403+0000 7f771ffff700 1 -- 172.21.15.123:0/2237820028 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7710014420 con 0x7f77280f5410 2024-01-13T11:57:25.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.403+0000 7f772e1ed700 1 -- 172.21.15.123:0/2237820028 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7728059100 con 0x7f77280f5410 2024-01-13T11:57:25.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.403+0000 7f771ffff700 1 --2- 172.21.15.123:0/2237820028 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f771402ff60 0x7f7714032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:25.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.403+0000 7f771ffff700 1 -- 172.21.15.123:0/2237820028 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7710041000 con 0x7f77280f5410 2024-01-13T11:57:25.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.403+0000 7f7726ffd700 1 --2- 172.21.15.123:0/2237820028 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f771402ff60 0x7f7714032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:25.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.404+0000 7f7726ffd700 1 --2- 172.21.15.123:0/2237820028 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f771402ff60 0x7f7714032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f77180096a0 tx=0x7f7718006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:25.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.408+0000 7f771ffff700 1 -- 172.21.15.123:0/2237820028 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f77100095f0 con 0x7f77280f5410 2024-01-13T11:57:25.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.613+0000 7f772e1ed700 1 -- 172.21.15.123:0/2237820028 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7728045ad0 con 0x7f77280f5410 2024-01-13T11:57:25.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.613+0000 7f771ffff700 1 -- 172.21.15.123:0/2237820028 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f771001db70 con 0x7f77280f5410 2024-01-13T11:57:25.601 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:25.601 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:25.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.615+0000 7f772e1ed700 1 -- 172.21.15.123:0/2237820028 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f771402ff60 msgr2=0x7f7714032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:25.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.615+0000 7f772e1ed700 1 --2- 172.21.15.123:0/2237820028 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f771402ff60 0x7f7714032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f77180096a0 tx=0x7f7718006b40).stop 2024-01-13T11:57:25.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.615+0000 7f772e1ed700 1 -- 172.21.15.123:0/2237820028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77280f5410 msgr2=0x7f7728109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:25.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.615+0000 7f772e1ed700 1 --2- 172.21.15.123:0/2237820028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77280f5410 0x7f7728109450 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f7710013510 tx=0x7f771000b040).stop 2024-01-13T11:57:25.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.615+0000 7f772e1ed700 1 -- 172.21.15.123:0/2237820028 shutdown_connections 2024-01-13T11:57:25.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.615+0000 7f772e1ed700 1 --2- 172.21.15.123:0/2237820028 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f771402ff60 0x7f7714032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:25.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.615+0000 7f772e1ed700 1 --2- 172.21.15.123:0/2237820028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77280f5410 0x7f7728109450 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:25.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.615+0000 7f772e1ed700 1 -- 172.21.15.123:0/2237820028 >> 172.21.15.123:0/2237820028 conn(0x7f77280f0b70 msgr2=0x7f77280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:25.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.615+0000 7f772e1ed700 1 -- 172.21.15.123:0/2237820028 shutdown_connections 2024-01-13T11:57:25.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:25.615+0000 7f772e1ed700 1 -- 172.21.15.123:0/2237820028 wait complete. 2024-01-13T11:57:25.603 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:25.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:25 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:23.574333+0000 mgr.smithi059.hmyldj (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:26.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:26 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:25.614791+0000 mon.smithi059 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.123:0/2237820028' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:27.019 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:27.019 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:27.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:27 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:25.574654+0000 mgr.smithi059.hmyldj (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:28.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.284+0000 7f552b59e700 1 -- 172.21.15.123:0/3564938441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f552c0f51a0 msgr2=0x7f552c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:28.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.284+0000 7f552b59e700 1 --2- 172.21.15.123:0/3564938441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f552c0f51a0 0x7f552c0f55c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f5520009fb0 tx=0x7f5520009b30).stop 2024-01-13T11:57:28.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.284+0000 7f552b59e700 1 -- 172.21.15.123:0/3564938441 shutdown_connections 2024-01-13T11:57:28.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.284+0000 7f552b59e700 1 --2- 172.21.15.123:0/3564938441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f552c0f51a0 0x7f552c0f55c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:28.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.284+0000 7f552b59e700 1 -- 172.21.15.123:0/3564938441 >> 172.21.15.123:0/3564938441 conn(0x7f552c0f09a0 msgr2=0x7f552c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:28.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.284+0000 7f552b59e700 1 -- 172.21.15.123:0/3564938441 shutdown_connections 2024-01-13T11:57:28.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.284+0000 7f552b59e700 1 -- 172.21.15.123:0/3564938441 wait complete. 2024-01-13T11:57:28.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.285+0000 7f552b59e700 1 Processor -- start 2024-01-13T11:57:28.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.285+0000 7f552b59e700 1 -- start start 2024-01-13T11:57:28.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.285+0000 7f552b59e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f552c0f51a0 0x7f552c0f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:28.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.285+0000 7f552b59e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552c0f13d0 con 0x7f552c0f51a0 2024-01-13T11:57:28.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.285+0000 7f552a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f552c0f51a0 0x7f552c0f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:28.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.285+0000 7f552a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f552c0f51a0 0x7f552c0f2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:56596/0 (socket says 172.21.15.123:56596) 2024-01-13T11:57:28.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.285+0000 7f552a59c700 1 -- 172.21.15.123:0/12038606 learned_addr learned my addr 172.21.15.123:0/12038606 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:28.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.286+0000 7f552a59c700 1 -- 172.21.15.123:0/12038606 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5520005040 con 0x7f552c0f51a0 2024-01-13T11:57:28.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.286+0000 7f552a59c700 1 --2- 172.21.15.123:0/12038606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f552c0f51a0 0x7f552c0f2f50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f5520013510 tx=0x7f552000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:28.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.286+0000 7f551affd700 1 -- 172.21.15.123:0/12038606 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5520009320 con 0x7f552c0f51a0 2024-01-13T11:57:28.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.286+0000 7f552b59e700 1 -- 172.21.15.123:0/12038606 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f552c0f15d0 con 0x7f552c0f51a0 2024-01-13T11:57:28.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.286+0000 7f551affd700 1 -- 172.21.15.123:0/12038606 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5520013f00 con 0x7f552c0f51a0 2024-01-13T11:57:28.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.286+0000 7f551affd700 1 -- 172.21.15.123:0/12038606 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5520014dc0 con 0x7f552c0f51a0 2024-01-13T11:57:28.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.286+0000 7f552b59e700 1 -- 172.21.15.123:0/12038606 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f552c0f1a60 con 0x7f552c0f51a0 2024-01-13T11:57:28.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.287+0000 7f552b59e700 1 -- 172.21.15.123:0/12038606 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f552c059100 con 0x7f552c0f51a0 2024-01-13T11:57:28.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.287+0000 7f551affd700 1 -- 172.21.15.123:0/12038606 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f55200095f0 con 0x7f552c0f51a0 2024-01-13T11:57:28.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.287+0000 7f551affd700 1 --2- 172.21.15.123:0/12038606 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f551402ff60 0x7f5514032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:28.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.287+0000 7f551affd700 1 -- 172.21.15.123:0/12038606 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5520041020 con 0x7f552c0f51a0 2024-01-13T11:57:28.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.287+0000 7f5529d9b700 1 --2- 172.21.15.123:0/12038606 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f551402ff60 0x7f5514032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:28.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.288+0000 7f5529d9b700 1 --2- 172.21.15.123:0/12038606 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f551402ff60 0x7f5514032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f551c0096a0 tx=0x7f551c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:28.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.291+0000 7f551affd700 1 -- 172.21.15.123:0/12038606 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5520014420 con 0x7f552c0f51a0 2024-01-13T11:57:28.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.496+0000 7f552b59e700 1 -- 172.21.15.123:0/12038606 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f552c027a00 con 0x7f552c0f51a0 2024-01-13T11:57:28.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.498+0000 7f551affd700 1 -- 172.21.15.123:0/12038606 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f552001c920 con 0x7f552c0f51a0 2024-01-13T11:57:28.484 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:28.484 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:28.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.498+0000 7f552b59e700 1 -- 172.21.15.123:0/12038606 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f551402ff60 msgr2=0x7f5514032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:28.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.498+0000 7f552b59e700 1 --2- 172.21.15.123:0/12038606 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f551402ff60 0x7f5514032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f551c0096a0 tx=0x7f551c006b40).stop 2024-01-13T11:57:28.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.498+0000 7f552b59e700 1 -- 172.21.15.123:0/12038606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f552c0f51a0 msgr2=0x7f552c0f2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:28.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.498+0000 7f552b59e700 1 --2- 172.21.15.123:0/12038606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f552c0f51a0 0x7f552c0f2f50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f5520013510 tx=0x7f552000b040).stop 2024-01-13T11:57:28.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.498+0000 7f552b59e700 1 -- 172.21.15.123:0/12038606 shutdown_connections 2024-01-13T11:57:28.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.498+0000 7f552b59e700 1 --2- 172.21.15.123:0/12038606 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f551402ff60 0x7f5514032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:28.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.498+0000 7f552b59e700 1 --2- 172.21.15.123:0/12038606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f552c0f51a0 0x7f552c0f2f50 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:28.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.498+0000 7f552b59e700 1 -- 172.21.15.123:0/12038606 >> 172.21.15.123:0/12038606 conn(0x7f552c0f09a0 msgr2=0x7f552c05fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:28.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.498+0000 7f552b59e700 1 -- 172.21.15.123:0/12038606 shutdown_connections 2024-01-13T11:57:28.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:28.498+0000 7f552b59e700 1 -- 172.21.15.123:0/12038606 wait complete. 2024-01-13T11:57:28.487 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:28.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:28 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:27.574971+0000 mgr.smithi059.hmyldj (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:29.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:29 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:28.498093+0000 mon.smithi059 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.123:0/12038606' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:29.875 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:29.876 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:30.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:30 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:29.575282+0000 mgr.smithi059.hmyldj (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:31.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.086+0000 7fd41e2cd700 1 -- 172.21.15.123:0/564261672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd418060280 msgr2=0x7fd4180f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:31.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.086+0000 7fd41e2cd700 1 --2- 172.21.15.123:0/564261672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd418060280 0x7fd4180f90f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fd408009fb0 tx=0x7fd408009b30).stop 2024-01-13T11:57:31.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.086+0000 7fd41e2cd700 1 -- 172.21.15.123:0/564261672 shutdown_connections 2024-01-13T11:57:31.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.086+0000 7fd41e2cd700 1 --2- 172.21.15.123:0/564261672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd418060280 0x7fd4180f90f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:31.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.086+0000 7fd41e2cd700 1 -- 172.21.15.123:0/564261672 >> 172.21.15.123:0/564261672 conn(0x7fd4180f0b70 msgr2=0x7fd4180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:31.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.086+0000 7fd41e2cd700 1 -- 172.21.15.123:0/564261672 shutdown_connections 2024-01-13T11:57:31.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.087+0000 7fd41e2cd700 1 -- 172.21.15.123:0/564261672 wait complete. 2024-01-13T11:57:31.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.087+0000 7fd41e2cd700 1 Processor -- start 2024-01-13T11:57:31.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.087+0000 7fd41e2cd700 1 -- start start 2024-01-13T11:57:31.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.088+0000 7fd41e2cd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd418060280 0x7fd4181066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:31.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.088+0000 7fd41e2cd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd418106bd0 con 0x7fd418060280 2024-01-13T11:57:31.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.088+0000 7fd417fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd418060280 0x7fd4181066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:31.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.088+0000 7fd417fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd418060280 0x7fd4181066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:56616/0 (socket says 172.21.15.123:56616) 2024-01-13T11:57:31.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.088+0000 7fd417fff700 1 -- 172.21.15.123:0/2543975504 learned_addr learned my addr 172.21.15.123:0/2543975504 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:31.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.088+0000 7fd417fff700 1 -- 172.21.15.123:0/2543975504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd408005040 con 0x7fd418060280 2024-01-13T11:57:31.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.088+0000 7fd417fff700 1 --2- 172.21.15.123:0/2543975504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd418060280 0x7fd4181066e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fd408013510 tx=0x7fd40800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:31.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.088+0000 7fd414ff9700 1 -- 172.21.15.123:0/2543975504 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd408009320 con 0x7fd418060280 2024-01-13T11:57:31.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.088+0000 7fd41e2cd700 1 -- 172.21.15.123:0/2543975504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd418106dd0 con 0x7fd418060280 2024-01-13T11:57:31.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.088+0000 7fd414ff9700 1 -- 172.21.15.123:0/2543975504 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd408013f00 con 0x7fd418060280 2024-01-13T11:57:31.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.088+0000 7fd414ff9700 1 -- 172.21.15.123:0/2543975504 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd408014dc0 con 0x7fd418060280 2024-01-13T11:57:31.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.088+0000 7fd41e2cd700 1 -- 172.21.15.123:0/2543975504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd418107260 con 0x7fd418060280 2024-01-13T11:57:31.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.089+0000 7fd414ff9700 1 -- 172.21.15.123:0/2543975504 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd40801d3d0 con 0x7fd418060280 2024-01-13T11:57:31.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.089+0000 7fd414ff9700 1 --2- 172.21.15.123:0/2543975504 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd40002ff60 0x7fd400032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:31.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.089+0000 7fd414ff9700 1 -- 172.21.15.123:0/2543975504 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd408041020 con 0x7fd418060280 2024-01-13T11:57:31.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.089+0000 7fd41e2cd700 1 -- 172.21.15.123:0/2543975504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd418045ad0 con 0x7fd418060280 2024-01-13T11:57:31.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.090+0000 7fd4177fe700 1 --2- 172.21.15.123:0/2543975504 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd40002ff60 0x7fd400032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:31.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.090+0000 7fd4177fe700 1 --2- 172.21.15.123:0/2543975504 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd40002ff60 0x7fd400032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fd40c0096a0 tx=0x7fd40c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:31.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.094+0000 7fd414ff9700 1 -- 172.21.15.123:0/2543975504 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd408012900 con 0x7fd418060280 2024-01-13T11:57:31.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.297+0000 7fd41e2cd700 1 -- 172.21.15.123:0/2543975504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd418108020 con 0x7fd418060280 2024-01-13T11:57:31.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.298+0000 7fd414ff9700 1 -- 172.21.15.123:0/2543975504 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd40802b020 con 0x7fd418060280 2024-01-13T11:57:31.284 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:31.285 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:31.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.299+0000 7fd41e2cd700 1 -- 172.21.15.123:0/2543975504 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd40002ff60 msgr2=0x7fd400032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:31.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.299+0000 7fd41e2cd700 1 --2- 172.21.15.123:0/2543975504 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd40002ff60 0x7fd400032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fd40c0096a0 tx=0x7fd40c006b40).stop 2024-01-13T11:57:31.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.299+0000 7fd41e2cd700 1 -- 172.21.15.123:0/2543975504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd418060280 msgr2=0x7fd4181066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:31.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.299+0000 7fd41e2cd700 1 --2- 172.21.15.123:0/2543975504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd418060280 0x7fd4181066e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fd408013510 tx=0x7fd40800b040).stop 2024-01-13T11:57:31.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.299+0000 7fd41e2cd700 1 -- 172.21.15.123:0/2543975504 shutdown_connections 2024-01-13T11:57:31.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.299+0000 7fd41e2cd700 1 --2- 172.21.15.123:0/2543975504 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fd40002ff60 0x7fd400032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:31.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.299+0000 7fd41e2cd700 1 --2- 172.21.15.123:0/2543975504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd418060280 0x7fd4181066e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:31.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.299+0000 7fd41e2cd700 1 -- 172.21.15.123:0/2543975504 >> 172.21.15.123:0/2543975504 conn(0x7fd4180f0b70 msgr2=0x7fd4180f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:31.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.300+0000 7fd41e2cd700 1 -- 172.21.15.123:0/2543975504 shutdown_connections 2024-01-13T11:57:31.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:31.300+0000 7fd41e2cd700 1 -- 172.21.15.123:0/2543975504 wait complete. 2024-01-13T11:57:31.287 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:32.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:32 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:31.298301+0000 mon.smithi059 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.123:0/2543975504' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:32.681 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:32.682 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:33.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:33 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:31.575601+0000 mgr.smithi059.hmyldj (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:33.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.837+0000 7f6351cba700 1 -- 172.21.15.123:0/3321258711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f634c0f51a0 msgr2=0x7f634c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:33.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.837+0000 7f6351cba700 1 --2- 172.21.15.123:0/3321258711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f634c0f51a0 0x7f634c0f55c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f6340009fb0 tx=0x7f6340009b30).stop 2024-01-13T11:57:33.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.837+0000 7f6351cba700 1 -- 172.21.15.123:0/3321258711 shutdown_connections 2024-01-13T11:57:33.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.837+0000 7f6351cba700 1 --2- 172.21.15.123:0/3321258711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f634c0f51a0 0x7f634c0f55c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:33.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.837+0000 7f6351cba700 1 -- 172.21.15.123:0/3321258711 >> 172.21.15.123:0/3321258711 conn(0x7f634c0f09a0 msgr2=0x7f634c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:33.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.837+0000 7f6351cba700 1 -- 172.21.15.123:0/3321258711 shutdown_connections 2024-01-13T11:57:33.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.837+0000 7f6351cba700 1 -- 172.21.15.123:0/3321258711 wait complete. 2024-01-13T11:57:33.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.837+0000 7f6351cba700 1 Processor -- start 2024-01-13T11:57:33.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.837+0000 7f6351cba700 1 -- start start 2024-01-13T11:57:33.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.838+0000 7f6351cba700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f634c0f51a0 0x7f634c1064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:33.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.838+0000 7f6351cba700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634c106990 con 0x7f634c0f51a0 2024-01-13T11:57:33.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.838+0000 7f6350cb8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f634c0f51a0 0x7f634c1064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:33.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.838+0000 7f6350cb8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f634c0f51a0 0x7f634c1064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:56632/0 (socket says 172.21.15.123:56632) 2024-01-13T11:57:33.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.838+0000 7f6350cb8700 1 -- 172.21.15.123:0/2376313504 learned_addr learned my addr 172.21.15.123:0/2376313504 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:33.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.838+0000 7f6350cb8700 1 -- 172.21.15.123:0/2376313504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6340005040 con 0x7f634c0f51a0 2024-01-13T11:57:33.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.838+0000 7f6350cb8700 1 --2- 172.21.15.123:0/2376313504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f634c0f51a0 0x7f634c1064a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f6340013510 tx=0x7f634000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:33.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.839+0000 7f63497fa700 1 -- 172.21.15.123:0/2376313504 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6340009320 con 0x7f634c0f51a0 2024-01-13T11:57:33.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.839+0000 7f6351cba700 1 -- 172.21.15.123:0/2376313504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f634c106b90 con 0x7f634c0f51a0 2024-01-13T11:57:33.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.839+0000 7f63497fa700 1 -- 172.21.15.123:0/2376313504 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6340013f00 con 0x7f634c0f51a0 2024-01-13T11:57:33.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.839+0000 7f63497fa700 1 -- 172.21.15.123:0/2376313504 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6340014dc0 con 0x7f634c0f51a0 2024-01-13T11:57:33.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.839+0000 7f6351cba700 1 -- 172.21.15.123:0/2376313504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f634c107020 con 0x7f634c0f51a0 2024-01-13T11:57:33.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.839+0000 7f6351cba700 1 -- 172.21.15.123:0/2376313504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f634c045ad0 con 0x7f634c0f51a0 2024-01-13T11:57:33.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.839+0000 7f63497fa700 1 -- 172.21.15.123:0/2376313504 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6340014420 con 0x7f634c0f51a0 2024-01-13T11:57:33.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.840+0000 7f63497fa700 1 --2- 172.21.15.123:0/2376313504 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f633c02ff60 0x7f633c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:33.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.840+0000 7f63497fa700 1 -- 172.21.15.123:0/2376313504 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6340041050 con 0x7f634c0f51a0 2024-01-13T11:57:33.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.840+0000 7f634bfff700 1 --2- 172.21.15.123:0/2376313504 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f633c02ff60 0x7f633c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:33.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.840+0000 7f634bfff700 1 --2- 172.21.15.123:0/2376313504 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f633c02ff60 0x7f633c032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f63380096a0 tx=0x7f6338006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:33.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:33.844+0000 7f63497fa700 1 -- 172.21.15.123:0/2376313504 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63400095f0 con 0x7f634c0f51a0 2024-01-13T11:57:34.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:34.051+0000 7f6351cba700 1 -- 172.21.15.123:0/2376313504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f634c107ed0 con 0x7f634c0f51a0 2024-01-13T11:57:34.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:34.052+0000 7f63497fa700 1 -- 172.21.15.123:0/2376313504 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f634001cb10 con 0x7f634c0f51a0 2024-01-13T11:57:34.039 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:34.039 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:34.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:34.053+0000 7f6351cba700 1 -- 172.21.15.123:0/2376313504 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f633c02ff60 msgr2=0x7f633c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:34.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:34.053+0000 7f6351cba700 1 --2- 172.21.15.123:0/2376313504 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f633c02ff60 0x7f633c032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f63380096a0 tx=0x7f6338006b40).stop 2024-01-13T11:57:34.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:34.053+0000 7f6351cba700 1 -- 172.21.15.123:0/2376313504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f634c0f51a0 msgr2=0x7f634c1064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:34.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:34.053+0000 7f6351cba700 1 --2- 172.21.15.123:0/2376313504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f634c0f51a0 0x7f634c1064a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f6340013510 tx=0x7f634000b040).stop 2024-01-13T11:57:34.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:34.053+0000 7f6351cba700 1 -- 172.21.15.123:0/2376313504 shutdown_connections 2024-01-13T11:57:34.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:34.053+0000 7f6351cba700 1 --2- 172.21.15.123:0/2376313504 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f633c02ff60 0x7f633c032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:34.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:34.053+0000 7f6351cba700 1 --2- 172.21.15.123:0/2376313504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f634c0f51a0 0x7f634c1064a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:34.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:34.054+0000 7f6351cba700 1 -- 172.21.15.123:0/2376313504 >> 172.21.15.123:0/2376313504 conn(0x7f634c0f09a0 msgr2=0x7f634c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:34.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:34.054+0000 7f6351cba700 1 -- 172.21.15.123:0/2376313504 shutdown_connections 2024-01-13T11:57:34.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:34.054+0000 7f6351cba700 1 -- 172.21.15.123:0/2376313504 wait complete. 2024-01-13T11:57:34.041 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:34.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:34 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:34.052316+0000 mon.smithi059 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.123:0/2376313504' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:35.412 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:35.412 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:35.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:35 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:33.575965+0000 mgr.smithi059.hmyldj (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:36.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.640+0000 7f8e16fa7700 1 -- 172.21.15.123:0/4268641663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e100f5410 msgr2=0x7f8e100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:36.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.640+0000 7f8e16fa7700 1 --2- 172.21.15.123:0/4268641663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e100f5410 0x7f8e100f5830 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f8e04009fb0 tx=0x7f8e04009b30).stop 2024-01-13T11:57:36.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.640+0000 7f8e16fa7700 1 -- 172.21.15.123:0/4268641663 shutdown_connections 2024-01-13T11:57:36.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.640+0000 7f8e16fa7700 1 --2- 172.21.15.123:0/4268641663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e100f5410 0x7f8e100f5830 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:36.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.640+0000 7f8e16fa7700 1 -- 172.21.15.123:0/4268641663 >> 172.21.15.123:0/4268641663 conn(0x7f8e100f0b70 msgr2=0x7f8e100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:36.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.640+0000 7f8e16fa7700 1 -- 172.21.15.123:0/4268641663 shutdown_connections 2024-01-13T11:57:36.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.640+0000 7f8e16fa7700 1 -- 172.21.15.123:0/4268641663 wait complete. 2024-01-13T11:57:36.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.641+0000 7f8e16fa7700 1 Processor -- start 2024-01-13T11:57:36.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.641+0000 7f8e16fa7700 1 -- start start 2024-01-13T11:57:36.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.641+0000 7f8e16fa7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e100f5410 0x7f8e10105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:36.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.641+0000 7f8e16fa7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e10105af0 con 0x7f8e100f5410 2024-01-13T11:57:36.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.642+0000 7f8e14d43700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e100f5410 0x7f8e10105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:36.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.642+0000 7f8e14d43700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e100f5410 0x7f8e10105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:37622/0 (socket says 172.21.15.123:37622) 2024-01-13T11:57:36.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.642+0000 7f8e14d43700 1 -- 172.21.15.123:0/180036776 learned_addr learned my addr 172.21.15.123:0/180036776 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:36.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.642+0000 7f8e14d43700 1 -- 172.21.15.123:0/180036776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e04005040 con 0x7f8e100f5410 2024-01-13T11:57:36.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.642+0000 7f8e14d43700 1 --2- 172.21.15.123:0/180036776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e100f5410 0x7f8e10105600 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f8e04013510 tx=0x7f8e0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:36.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.642+0000 7f8e0d7fa700 1 -- 172.21.15.123:0/180036776 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e04009320 con 0x7f8e100f5410 2024-01-13T11:57:36.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.642+0000 7f8e16fa7700 1 -- 172.21.15.123:0/180036776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e10105cf0 con 0x7f8e100f5410 2024-01-13T11:57:36.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.642+0000 7f8e0d7fa700 1 -- 172.21.15.123:0/180036776 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e04013f00 con 0x7f8e100f5410 2024-01-13T11:57:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.642+0000 7f8e0d7fa700 1 -- 172.21.15.123:0/180036776 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e04014d90 con 0x7f8e100f5410 2024-01-13T11:57:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.642+0000 7f8e16fa7700 1 -- 172.21.15.123:0/180036776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e10106180 con 0x7f8e100f5410 2024-01-13T11:57:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.643+0000 7f8e0d7fa700 1 -- 172.21.15.123:0/180036776 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8e040263a0 con 0x7f8e100f5410 2024-01-13T11:57:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.643+0000 7f8e0d7fa700 1 --2- 172.21.15.123:0/180036776 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8e0002ff60 0x7f8e00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.643+0000 7f8e0d7fa700 1 -- 172.21.15.123:0/180036776 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8e040410e0 con 0x7f8e100f5410 2024-01-13T11:57:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.643+0000 7f8e16fa7700 1 -- 172.21.15.123:0/180036776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e10045ad0 con 0x7f8e100f5410 2024-01-13T11:57:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.643+0000 7f8e0ffff700 1 --2- 172.21.15.123:0/180036776 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8e0002ff60 0x7f8e00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:36.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.644+0000 7f8e0ffff700 1 --2- 172.21.15.123:0/180036776 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8e0002ff60 0x7f8e00032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f8dfc0096a0 tx=0x7f8dfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:36.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.648+0000 7f8e0d7fa700 1 -- 172.21.15.123:0/180036776 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8e04012920 con 0x7f8e100f5410 2024-01-13T11:57:36.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.852+0000 7f8e16fa7700 1 -- 172.21.15.123:0/180036776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e100faf50 con 0x7f8e100f5410 2024-01-13T11:57:36.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.853+0000 7f8e0d7fa700 1 -- 172.21.15.123:0/180036776 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8e0402b020 con 0x7f8e100f5410 2024-01-13T11:57:36.844 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:36.844 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:36.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.855+0000 7f8e16fa7700 1 -- 172.21.15.123:0/180036776 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8e0002ff60 msgr2=0x7f8e00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:36.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.855+0000 7f8e16fa7700 1 --2- 172.21.15.123:0/180036776 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8e0002ff60 0x7f8e00032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f8dfc0096a0 tx=0x7f8dfc006b40).stop 2024-01-13T11:57:36.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.855+0000 7f8e16fa7700 1 -- 172.21.15.123:0/180036776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e100f5410 msgr2=0x7f8e10105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:36.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.855+0000 7f8e16fa7700 1 --2- 172.21.15.123:0/180036776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e100f5410 0x7f8e10105600 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f8e04013510 tx=0x7f8e0400b040).stop 2024-01-13T11:57:36.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.855+0000 7f8e16fa7700 1 -- 172.21.15.123:0/180036776 shutdown_connections 2024-01-13T11:57:36.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.855+0000 7f8e16fa7700 1 --2- 172.21.15.123:0/180036776 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8e0002ff60 0x7f8e00032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:36.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.855+0000 7f8e16fa7700 1 --2- 172.21.15.123:0/180036776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e100f5410 0x7f8e10105600 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:36.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.855+0000 7f8e16fa7700 1 -- 172.21.15.123:0/180036776 >> 172.21.15.123:0/180036776 conn(0x7f8e100f0b70 msgr2=0x7f8e100f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:36.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.855+0000 7f8e16fa7700 1 -- 172.21.15.123:0/180036776 shutdown_connections 2024-01-13T11:57:36.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:36.855+0000 7f8e16fa7700 1 -- 172.21.15.123:0/180036776 wait complete. 2024-01-13T11:57:36.845 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:37.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:37 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:35.576344+0000 mgr.smithi059.hmyldj (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:37.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:37 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:36.854527+0000 mon.smithi059 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.123:0/180036776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:38.220 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:38.221 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:39.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.447+0000 7f359b59e700 1 -- 172.21.15.123:0/464719613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f359c0f51a0 msgr2=0x7f359c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:39.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.447+0000 7f359b59e700 1 --2- 172.21.15.123:0/464719613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f359c0f51a0 0x7f359c0f55c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f358c009fb0 tx=0x7f358c009b30).stop 2024-01-13T11:57:39.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.447+0000 7f359b59e700 1 -- 172.21.15.123:0/464719613 shutdown_connections 2024-01-13T11:57:39.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.447+0000 7f359b59e700 1 --2- 172.21.15.123:0/464719613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f359c0f51a0 0x7f359c0f55c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:39.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.447+0000 7f359b59e700 1 -- 172.21.15.123:0/464719613 >> 172.21.15.123:0/464719613 conn(0x7f359c0f09a0 msgr2=0x7f359c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:39.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.447+0000 7f359b59e700 1 -- 172.21.15.123:0/464719613 shutdown_connections 2024-01-13T11:57:39.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.447+0000 7f359b59e700 1 -- 172.21.15.123:0/464719613 wait complete. 2024-01-13T11:57:39.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.447+0000 7f359b59e700 1 Processor -- start 2024-01-13T11:57:39.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.447+0000 7f359b59e700 1 -- start start 2024-01-13T11:57:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.447+0000 7f359b59e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f359c0f51a0 0x7f359c1064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.447+0000 7f359b59e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c1069b0 con 0x7f359c0f51a0 2024-01-13T11:57:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.448+0000 7f359a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f359c0f51a0 0x7f359c1064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.448+0000 7f359a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f359c0f51a0 0x7f359c1064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:37644/0 (socket says 172.21.15.123:37644) 2024-01-13T11:57:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.448+0000 7f359a59c700 1 -- 172.21.15.123:0/2538980428 learned_addr learned my addr 172.21.15.123:0/2538980428 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.448+0000 7f359a59c700 1 -- 172.21.15.123:0/2538980428 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c005040 con 0x7f359c0f51a0 2024-01-13T11:57:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.448+0000 7f359a59c700 1 --2- 172.21.15.123:0/2538980428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f359c0f51a0 0x7f359c1064c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f358c009830 tx=0x7f358c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.448+0000 7f358affd700 1 -- 172.21.15.123:0/2538980428 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f358c009320 con 0x7f359c0f51a0 2024-01-13T11:57:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.448+0000 7f358affd700 1 -- 172.21.15.123:0/2538980428 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f358c0136c0 con 0x7f359c0f51a0 2024-01-13T11:57:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.448+0000 7f359b59e700 1 -- 172.21.15.123:0/2538980428 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359c106bb0 con 0x7f359c0f51a0 2024-01-13T11:57:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.448+0000 7f358affd700 1 -- 172.21.15.123:0/2538980428 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f358c01ee90 con 0x7f359c0f51a0 2024-01-13T11:57:39.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.448+0000 7f359b59e700 1 -- 172.21.15.123:0/2538980428 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359c107040 con 0x7f359c0f51a0 2024-01-13T11:57:39.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.449+0000 7f358affd700 1 -- 172.21.15.123:0/2538980428 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f358c01e420 con 0x7f359c0f51a0 2024-01-13T11:57:39.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.449+0000 7f359b59e700 1 -- 172.21.15.123:0/2538980428 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359c045ad0 con 0x7f359c0f51a0 2024-01-13T11:57:39.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.449+0000 7f358affd700 1 --2- 172.21.15.123:0/2538980428 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f358402ff10 0x7f35840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:39.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.449+0000 7f358affd700 1 -- 172.21.15.123:0/2538980428 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f358c040f40 con 0x7f359c0f51a0 2024-01-13T11:57:39.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.450+0000 7f3599d9b700 1 --2- 172.21.15.123:0/2538980428 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f358402ff10 0x7f35840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:39.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.450+0000 7f3599d9b700 1 --2- 172.21.15.123:0/2538980428 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f358402ff10 0x7f35840323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f35900096a0 tx=0x7f3590006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:39.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.454+0000 7f358affd700 1 -- 172.21.15.123:0/2538980428 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f358c06d2b0 con 0x7f359c0f51a0 2024-01-13T11:57:39.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:39 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:37.576673+0000 mgr.smithi059.hmyldj (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:39.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.658+0000 7f359b59e700 1 -- 172.21.15.123:0/2538980428 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f359c107ef0 con 0x7f359c0f51a0 2024-01-13T11:57:39.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.659+0000 7f358affd700 1 -- 172.21.15.123:0/2538980428 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f358c0095f0 con 0x7f359c0f51a0 2024-01-13T11:57:39.646 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:39.647 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:39.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.660+0000 7f359b59e700 1 -- 172.21.15.123:0/2538980428 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f358402ff10 msgr2=0x7f35840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:39.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.660+0000 7f359b59e700 1 --2- 172.21.15.123:0/2538980428 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f358402ff10 0x7f35840323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f35900096a0 tx=0x7f3590006b40).stop 2024-01-13T11:57:39.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.660+0000 7f359b59e700 1 -- 172.21.15.123:0/2538980428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f359c0f51a0 msgr2=0x7f359c1064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:39.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.660+0000 7f359b59e700 1 --2- 172.21.15.123:0/2538980428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f359c0f51a0 0x7f359c1064c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f358c009830 tx=0x7f358c00b040).stop 2024-01-13T11:57:39.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.660+0000 7f359b59e700 1 -- 172.21.15.123:0/2538980428 shutdown_connections 2024-01-13T11:57:39.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.660+0000 7f359b59e700 1 --2- 172.21.15.123:0/2538980428 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f358402ff10 0x7f35840323d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:39.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.660+0000 7f359b59e700 1 --2- 172.21.15.123:0/2538980428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f359c0f51a0 0x7f359c1064c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:39.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.660+0000 7f359b59e700 1 -- 172.21.15.123:0/2538980428 >> 172.21.15.123:0/2538980428 conn(0x7f359c0f09a0 msgr2=0x7f359c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:39.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.661+0000 7f359b59e700 1 -- 172.21.15.123:0/2538980428 shutdown_connections 2024-01-13T11:57:39.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:39.661+0000 7f359b59e700 1 -- 172.21.15.123:0/2538980428 wait complete. 2024-01-13T11:57:39.649 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:40.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:39.645660+0000 mon.smithi059 (mon.0) 300 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/mirror_snapshot_schedule"}]: dispatch 2024-01-13T11:57:40.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:39.645884+0000 mon.smithi059 (mon.0) 301 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/trash_purge_schedule"}]: dispatch 2024-01-13T11:57:40.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:39.660452+0000 mon.smithi059 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.123:0/2538980428' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:41.039 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:41.039 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:41.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:41 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:39.576972+0000 mgr.smithi059.hmyldj (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:42.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.235+0000 7f7addb9a700 1 -- 172.21.15.123:0/2724310853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad80f5950 msgr2=0x7f7ad80f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:42.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.235+0000 7f7addb9a700 1 --2- 172.21.15.123:0/2724310853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad80f5950 0x7f7ad80f5d70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f7ac0009fb0 tx=0x7f7ac0009b30).stop 2024-01-13T11:57:42.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.235+0000 7f7addb9a700 1 -- 172.21.15.123:0/2724310853 shutdown_connections 2024-01-13T11:57:42.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.235+0000 7f7addb9a700 1 --2- 172.21.15.123:0/2724310853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad80f5950 0x7f7ad80f5d70 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:42.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.235+0000 7f7addb9a700 1 -- 172.21.15.123:0/2724310853 >> 172.21.15.123:0/2724310853 conn(0x7f7ad80f0260 msgr2=0x7f7ad80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:42.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.235+0000 7f7addb9a700 1 -- 172.21.15.123:0/2724310853 shutdown_connections 2024-01-13T11:57:42.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.235+0000 7f7addb9a700 1 -- 172.21.15.123:0/2724310853 wait complete. 2024-01-13T11:57:42.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.236+0000 7f7addb9a700 1 Processor -- start 2024-01-13T11:57:42.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.236+0000 7f7addb9a700 1 -- start start 2024-01-13T11:57:42.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.236+0000 7f7addb9a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad8106800 0x7f7ad8106c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:42.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.236+0000 7f7addb9a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad8107110 con 0x7f7ad8106800 2024-01-13T11:57:42.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.236+0000 7f7ad77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad8106800 0x7f7ad8106c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:42.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.236+0000 7f7ad77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad8106800 0x7f7ad8106c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:37658/0 (socket says 172.21.15.123:37658) 2024-01-13T11:57:42.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.236+0000 7f7ad77fe700 1 -- 172.21.15.123:0/3320136246 learned_addr learned my addr 172.21.15.123:0/3320136246 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:42.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.237+0000 7f7ad77fe700 1 -- 172.21.15.123:0/3320136246 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac0005040 con 0x7f7ad8106800 2024-01-13T11:57:42.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.237+0000 7f7ad77fe700 1 --2- 172.21.15.123:0/3320136246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad8106800 0x7f7ad8106c20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f7ac0009f80 tx=0x7f7ac0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:42.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.237+0000 7f7acffff700 1 -- 172.21.15.123:0/3320136246 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ac00136c0 con 0x7f7ad8106800 2024-01-13T11:57:42.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.237+0000 7f7acffff700 1 -- 172.21.15.123:0/3320136246 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ac00095f0 con 0x7f7ad8106800 2024-01-13T11:57:42.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.237+0000 7f7addb9a700 1 -- 172.21.15.123:0/3320136246 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad8107310 con 0x7f7ad8106800 2024-01-13T11:57:42.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.237+0000 7f7acffff700 1 -- 172.21.15.123:0/3320136246 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ac0014420 con 0x7f7ad8106800 2024-01-13T11:57:42.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.237+0000 7f7addb9a700 1 -- 172.21.15.123:0/3320136246 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad8107e60 con 0x7f7ad8106800 2024-01-13T11:57:42.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.238+0000 7f7acffff700 1 -- 172.21.15.123:0/3320136246 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ac0025060 con 0x7f7ad8106800 2024-01-13T11:57:42.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.238+0000 7f7addb9a700 1 -- 172.21.15.123:0/3320136246 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad8059100 con 0x7f7ad8106800 2024-01-13T11:57:42.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.238+0000 7f7acffff700 1 --2- 172.21.15.123:0/3320136246 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7ac402ff60 0x7f7ac4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:42.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.238+0000 7f7acffff700 1 -- 172.21.15.123:0/3320136246 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ac0040da0 con 0x7f7ad8106800 2024-01-13T11:57:42.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.238+0000 7f7ad6ffd700 1 --2- 172.21.15.123:0/3320136246 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7ac402ff60 0x7f7ac4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:42.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.238+0000 7f7ad6ffd700 1 --2- 172.21.15.123:0/3320136246 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7ac402ff60 0x7f7ac4032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f7ac80096a0 tx=0x7f7ac8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:42.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.243+0000 7f7acffff700 1 -- 172.21.15.123:0/3320136246 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ac001cb60 con 0x7f7ad8106800 2024-01-13T11:57:42.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.450+0000 7f7addb9a700 1 -- 172.21.15.123:0/3320136246 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ad8045ad0 con 0x7f7ad8106800 2024-01-13T11:57:42.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.451+0000 7f7acffff700 1 -- 172.21.15.123:0/3320136246 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7ac000f280 con 0x7f7ad8106800 2024-01-13T11:57:42.439 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:42.439 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:42.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.453+0000 7f7addb9a700 1 -- 172.21.15.123:0/3320136246 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7ac402ff60 msgr2=0x7f7ac4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:42.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.453+0000 7f7addb9a700 1 --2- 172.21.15.123:0/3320136246 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7ac402ff60 0x7f7ac4032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f7ac80096a0 tx=0x7f7ac8006b40).stop 2024-01-13T11:57:42.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.453+0000 7f7addb9a700 1 -- 172.21.15.123:0/3320136246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad8106800 msgr2=0x7f7ad8106c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:42.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.453+0000 7f7addb9a700 1 --2- 172.21.15.123:0/3320136246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad8106800 0x7f7ad8106c20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f7ac0009f80 tx=0x7f7ac0009b30).stop 2024-01-13T11:57:42.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.453+0000 7f7addb9a700 1 -- 172.21.15.123:0/3320136246 shutdown_connections 2024-01-13T11:57:42.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.453+0000 7f7addb9a700 1 --2- 172.21.15.123:0/3320136246 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7ac402ff60 0x7f7ac4032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:42.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.453+0000 7f7addb9a700 1 --2- 172.21.15.123:0/3320136246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad8106800 0x7f7ad8106c20 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:42.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.453+0000 7f7addb9a700 1 -- 172.21.15.123:0/3320136246 >> 172.21.15.123:0/3320136246 conn(0x7f7ad80f0260 msgr2=0x7f7ad80f4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:42.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.453+0000 7f7addb9a700 1 -- 172.21.15.123:0/3320136246 shutdown_connections 2024-01-13T11:57:42.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:42.453+0000 7f7addb9a700 1 -- 172.21.15.123:0/3320136246 wait complete. 2024-01-13T11:57:42.441 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:43.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:43 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:41.577301+0000 mgr.smithi059.hmyldj (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:43.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:43 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:42.452425+0000 mon.smithi059 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.123:0/3320136246' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:43.822 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:43.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:44.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.976+0000 7f7d450a4700 1 -- 172.21.15.123:0/2551686142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d400f6510 msgr2=0x7f7d400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:44.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.976+0000 7f7d450a4700 1 --2- 172.21.15.123:0/2551686142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d400f6510 0x7f7d400f6930 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f7d34009fb0 tx=0x7f7d34009b30).stop 2024-01-13T11:57:44.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.976+0000 7f7d450a4700 1 -- 172.21.15.123:0/2551686142 shutdown_connections 2024-01-13T11:57:44.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.976+0000 7f7d450a4700 1 --2- 172.21.15.123:0/2551686142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d400f6510 0x7f7d400f6930 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:44.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.976+0000 7f7d450a4700 1 -- 172.21.15.123:0/2551686142 >> 172.21.15.123:0/2551686142 conn(0x7f7d400f1c70 msgr2=0x7f7d400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:44.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.976+0000 7f7d450a4700 1 -- 172.21.15.123:0/2551686142 shutdown_connections 2024-01-13T11:57:44.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.976+0000 7f7d450a4700 1 -- 172.21.15.123:0/2551686142 wait complete. 2024-01-13T11:57:44.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.977+0000 7f7d450a4700 1 Processor -- start 2024-01-13T11:57:44.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.977+0000 7f7d450a4700 1 -- start start 2024-01-13T11:57:44.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.977+0000 7f7d450a4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d400f6510 0x7f7d40106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:44.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.977+0000 7f7d450a4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d40106c20 con 0x7f7d400f6510 2024-01-13T11:57:44.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.977+0000 7f7d3ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d400f6510 0x7f7d40106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:44.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.977+0000 7f7d3ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d400f6510 0x7f7d40106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:37676/0 (socket says 172.21.15.123:37676) 2024-01-13T11:57:44.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.977+0000 7f7d3ed9d700 1 -- 172.21.15.123:0/2956711660 learned_addr learned my addr 172.21.15.123:0/2956711660 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:44.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.978+0000 7f7d3ed9d700 1 -- 172.21.15.123:0/2956711660 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d34005040 con 0x7f7d400f6510 2024-01-13T11:57:44.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.978+0000 7f7d3ed9d700 1 --2- 172.21.15.123:0/2956711660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d400f6510 0x7f7d40106730 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f7d34013510 tx=0x7f7d3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:44.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.978+0000 7f7d277fe700 1 -- 172.21.15.123:0/2956711660 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d34009320 con 0x7f7d400f6510 2024-01-13T11:57:44.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.978+0000 7f7d450a4700 1 -- 172.21.15.123:0/2956711660 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d40106e20 con 0x7f7d400f6510 2024-01-13T11:57:44.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.978+0000 7f7d277fe700 1 -- 172.21.15.123:0/2956711660 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d34013f00 con 0x7f7d400f6510 2024-01-13T11:57:44.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.978+0000 7f7d277fe700 1 -- 172.21.15.123:0/2956711660 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d34014d90 con 0x7f7d400f6510 2024-01-13T11:57:44.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.978+0000 7f7d450a4700 1 -- 172.21.15.123:0/2956711660 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d401072b0 con 0x7f7d400f6510 2024-01-13T11:57:44.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.979+0000 7f7d277fe700 1 -- 172.21.15.123:0/2956711660 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7d340095f0 con 0x7f7d400f6510 2024-01-13T11:57:44.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.979+0000 7f7d450a4700 1 -- 172.21.15.123:0/2956711660 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d40045ad0 con 0x7f7d400f6510 2024-01-13T11:57:44.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.979+0000 7f7d277fe700 1 --2- 172.21.15.123:0/2956711660 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7d2802ff60 0x7f7d28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:44.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.979+0000 7f7d277fe700 1 -- 172.21.15.123:0/2956711660 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7d34041080 con 0x7f7d400f6510 2024-01-13T11:57:44.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.979+0000 7f7d3e59c700 1 --2- 172.21.15.123:0/2956711660 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7d2802ff60 0x7f7d28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:44.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.980+0000 7f7d3e59c700 1 --2- 172.21.15.123:0/2956711660 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7d2802ff60 0x7f7d28032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f7d300096a0 tx=0x7f7d30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:44.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:44.984+0000 7f7d277fe700 1 -- 172.21.15.123:0/2956711660 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d34014420 con 0x7f7d400f6510 2024-01-13T11:57:45.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:45.191+0000 7f7d450a4700 1 -- 172.21.15.123:0/2956711660 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d400fbf60 con 0x7f7d400f6510 2024-01-13T11:57:45.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:45.192+0000 7f7d277fe700 1 -- 172.21.15.123:0/2956711660 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7d3401d7f0 con 0x7f7d400f6510 2024-01-13T11:57:45.179 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:45.179 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:45.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:45.193+0000 7f7d450a4700 1 -- 172.21.15.123:0/2956711660 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7d2802ff60 msgr2=0x7f7d28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:45.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:45.193+0000 7f7d450a4700 1 --2- 172.21.15.123:0/2956711660 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7d2802ff60 0x7f7d28032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f7d300096a0 tx=0x7f7d30006b40).stop 2024-01-13T11:57:45.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:45.193+0000 7f7d450a4700 1 -- 172.21.15.123:0/2956711660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d400f6510 msgr2=0x7f7d40106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:45.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:45.193+0000 7f7d450a4700 1 --2- 172.21.15.123:0/2956711660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d400f6510 0x7f7d40106730 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f7d34013510 tx=0x7f7d3400b040).stop 2024-01-13T11:57:45.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:45.193+0000 7f7d450a4700 1 -- 172.21.15.123:0/2956711660 shutdown_connections 2024-01-13T11:57:45.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:45.193+0000 7f7d450a4700 1 --2- 172.21.15.123:0/2956711660 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f7d2802ff60 0x7f7d28032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:45.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:45.193+0000 7f7d450a4700 1 --2- 172.21.15.123:0/2956711660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d400f6510 0x7f7d40106730 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:45.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:45.193+0000 7f7d450a4700 1 -- 172.21.15.123:0/2956711660 >> 172.21.15.123:0/2956711660 conn(0x7f7d400f1c70 msgr2=0x7f7d400f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:45.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:45.193+0000 7f7d450a4700 1 -- 172.21.15.123:0/2956711660 shutdown_connections 2024-01-13T11:57:45.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:45.193+0000 7f7d450a4700 1 -- 172.21.15.123:0/2956711660 wait complete. 2024-01-13T11:57:45.182 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:45.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:45 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:43.577596+0000 mgr.smithi059.hmyldj (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:45.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:45 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:45.193209+0000 mon.smithi059 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.123:0/2956711660' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:46.561 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:46.561 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:47.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:47 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:45.577961+0000 mgr.smithi059.hmyldj (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:47.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.718+0000 7ff6ca067700 1 -- 172.21.15.123:0/3881172271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c4060200 msgr2=0x7ff6c40fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:47.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.718+0000 7ff6ca067700 1 --2- 172.21.15.123:0/3881172271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c4060200 0x7ff6c40fb570 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7ff6ac009fb0 tx=0x7ff6ac009b30).stop 2024-01-13T11:57:47.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.718+0000 7ff6ca067700 1 -- 172.21.15.123:0/3881172271 shutdown_connections 2024-01-13T11:57:47.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.718+0000 7ff6ca067700 1 --2- 172.21.15.123:0/3881172271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c4060200 0x7ff6c40fb570 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:47.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.718+0000 7ff6ca067700 1 -- 172.21.15.123:0/3881172271 >> 172.21.15.123:0/3881172271 conn(0x7ff6c40f0b70 msgr2=0x7ff6c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:47.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.718+0000 7ff6ca067700 1 -- 172.21.15.123:0/3881172271 shutdown_connections 2024-01-13T11:57:47.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.718+0000 7ff6ca067700 1 -- 172.21.15.123:0/3881172271 wait complete. 2024-01-13T11:57:47.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.719+0000 7ff6ca067700 1 Processor -- start 2024-01-13T11:57:47.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.719+0000 7ff6ca067700 1 -- start start 2024-01-13T11:57:47.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.719+0000 7ff6ca067700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c4060200 0x7ff6c41088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:47.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.719+0000 7ff6ca067700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c4108dc0 con 0x7ff6c4060200 2024-01-13T11:57:47.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.720+0000 7ff6c37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c4060200 0x7ff6c41088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:47.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.720+0000 7ff6c37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c4060200 0x7ff6c41088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:35640/0 (socket says 172.21.15.123:35640) 2024-01-13T11:57:47.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.720+0000 7ff6c37fe700 1 -- 172.21.15.123:0/73329184 learned_addr learned my addr 172.21.15.123:0/73329184 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:47.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.720+0000 7ff6c37fe700 1 -- 172.21.15.123:0/73329184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ac005040 con 0x7ff6c4060200 2024-01-13T11:57:47.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.720+0000 7ff6c37fe700 1 --2- 172.21.15.123:0/73329184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c4060200 0x7ff6c41088d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7ff6ac009830 tx=0x7ff6ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:47.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.720+0000 7ff6bbfff700 1 -- 172.21.15.123:0/73329184 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6ac009320 con 0x7ff6c4060200 2024-01-13T11:57:47.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.720+0000 7ff6ca067700 1 -- 172.21.15.123:0/73329184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c4108fc0 con 0x7ff6c4060200 2024-01-13T11:57:47.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.720+0000 7ff6bbfff700 1 -- 172.21.15.123:0/73329184 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff6ac0136c0 con 0x7ff6c4060200 2024-01-13T11:57:47.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.720+0000 7ff6bbfff700 1 -- 172.21.15.123:0/73329184 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6ac01ee90 con 0x7ff6c4060200 2024-01-13T11:57:47.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.720+0000 7ff6ca067700 1 -- 172.21.15.123:0/73329184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c4109450 con 0x7ff6c4060200 2024-01-13T11:57:47.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.721+0000 7ff6ca067700 1 -- 172.21.15.123:0/73329184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c4045ad0 con 0x7ff6c4060200 2024-01-13T11:57:47.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.721+0000 7ff6bbfff700 1 -- 172.21.15.123:0/73329184 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff6ac01e420 con 0x7ff6c4060200 2024-01-13T11:57:47.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.721+0000 7ff6bbfff700 1 --2- 172.21.15.123:0/73329184 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff6b002ff10 0x7ff6b00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:47.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.721+0000 7ff6bbfff700 1 -- 172.21.15.123:0/73329184 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff6ac040f40 con 0x7ff6c4060200 2024-01-13T11:57:47.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.722+0000 7ff6c2ffd700 1 --2- 172.21.15.123:0/73329184 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff6b002ff10 0x7ff6b00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:47.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.722+0000 7ff6c2ffd700 1 --2- 172.21.15.123:0/73329184 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff6b002ff10 0x7ff6b00323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7ff6b40096a0 tx=0x7ff6b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:47.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.726+0000 7ff6bbfff700 1 -- 172.21.15.123:0/73329184 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff6ac06d2b0 con 0x7ff6c4060200 2024-01-13T11:57:47.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.931+0000 7ff6ca067700 1 -- 172.21.15.123:0/73329184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff6c410a2a0 con 0x7ff6c4060200 2024-01-13T11:57:47.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.931+0000 7ff6bbfff700 1 -- 172.21.15.123:0/73329184 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff6ac0095f0 con 0x7ff6c4060200 2024-01-13T11:57:47.919 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:47.919 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:47.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.933+0000 7ff6ca067700 1 -- 172.21.15.123:0/73329184 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff6b002ff10 msgr2=0x7ff6b00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:47.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.933+0000 7ff6ca067700 1 --2- 172.21.15.123:0/73329184 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff6b002ff10 0x7ff6b00323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7ff6b40096a0 tx=0x7ff6b4006b40).stop 2024-01-13T11:57:47.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.933+0000 7ff6ca067700 1 -- 172.21.15.123:0/73329184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c4060200 msgr2=0x7ff6c41088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:47.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.933+0000 7ff6ca067700 1 --2- 172.21.15.123:0/73329184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c4060200 0x7ff6c41088d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7ff6ac009830 tx=0x7ff6ac00b040).stop 2024-01-13T11:57:47.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.933+0000 7ff6ca067700 1 -- 172.21.15.123:0/73329184 shutdown_connections 2024-01-13T11:57:47.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.933+0000 7ff6ca067700 1 --2- 172.21.15.123:0/73329184 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff6b002ff10 0x7ff6b00323d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:47.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.933+0000 7ff6ca067700 1 --2- 172.21.15.123:0/73329184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c4060200 0x7ff6c41088d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:47.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.933+0000 7ff6ca067700 1 -- 172.21.15.123:0/73329184 >> 172.21.15.123:0/73329184 conn(0x7ff6c40f0b70 msgr2=0x7ff6c40f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:47.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.933+0000 7ff6ca067700 1 -- 172.21.15.123:0/73329184 shutdown_connections 2024-01-13T11:57:47.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:47.933+0000 7ff6ca067700 1 -- 172.21.15.123:0/73329184 wait complete. 2024-01-13T11:57:47.921 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:48.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:48 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:47.932616+0000 mon.smithi059 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.123:0/73329184' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:49.308 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:49.308 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:49.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:49 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:47.578265+0000 mgr.smithi059.hmyldj (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:50.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.492+0000 7fa0d359e700 1 -- 172.21.15.123:0/122720082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0d40f62a0 msgr2=0x7fa0d40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:50.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.492+0000 7fa0d359e700 1 --2- 172.21.15.123:0/122720082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0d40f62a0 0x7fa0d40f66c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fa0c8009fb0 tx=0x7fa0c8009b30).stop 2024-01-13T11:57:50.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.492+0000 7fa0d359e700 1 -- 172.21.15.123:0/122720082 shutdown_connections 2024-01-13T11:57:50.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.492+0000 7fa0d359e700 1 --2- 172.21.15.123:0/122720082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0d40f62a0 0x7fa0d40f66c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:50.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.492+0000 7fa0d359e700 1 -- 172.21.15.123:0/122720082 >> 172.21.15.123:0/122720082 conn(0x7fa0d40f1aa0 msgr2=0x7fa0d40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:50.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.492+0000 7fa0d359e700 1 -- 172.21.15.123:0/122720082 shutdown_connections 2024-01-13T11:57:50.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.492+0000 7fa0d359e700 1 -- 172.21.15.123:0/122720082 wait complete. 2024-01-13T11:57:50.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.492+0000 7fa0d359e700 1 Processor -- start 2024-01-13T11:57:50.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.492+0000 7fa0d359e700 1 -- start start 2024-01-13T11:57:50.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.492+0000 7fa0d359e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0d40f62a0 0x7fa0d41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:50.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.492+0000 7fa0d359e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d41069a0 con 0x7fa0d40f62a0 2024-01-13T11:57:50.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.493+0000 7fa0d259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0d40f62a0 0x7fa0d41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:50.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.493+0000 7fa0d259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0d40f62a0 0x7fa0d41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:35656/0 (socket says 172.21.15.123:35656) 2024-01-13T11:57:50.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.493+0000 7fa0d259c700 1 -- 172.21.15.123:0/738316852 learned_addr learned my addr 172.21.15.123:0/738316852 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:50.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.493+0000 7fa0d259c700 1 -- 172.21.15.123:0/738316852 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c8005040 con 0x7fa0d40f62a0 2024-01-13T11:57:50.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.493+0000 7fa0d259c700 1 --2- 172.21.15.123:0/738316852 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0d40f62a0 0x7fa0d41064b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fa0c8013510 tx=0x7fa0c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:50.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.493+0000 7fa0c2ffd700 1 -- 172.21.15.123:0/738316852 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0c8009320 con 0x7fa0d40f62a0 2024-01-13T11:57:50.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.493+0000 7fa0d359e700 1 -- 172.21.15.123:0/738316852 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0d4106ba0 con 0x7fa0d40f62a0 2024-01-13T11:57:50.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.493+0000 7fa0c2ffd700 1 -- 172.21.15.123:0/738316852 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa0c8013f00 con 0x7fa0d40f62a0 2024-01-13T11:57:50.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.493+0000 7fa0c2ffd700 1 -- 172.21.15.123:0/738316852 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0c8014dc0 con 0x7fa0d40f62a0 2024-01-13T11:57:50.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.493+0000 7fa0d359e700 1 -- 172.21.15.123:0/738316852 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0d4107030 con 0x7fa0d40f62a0 2024-01-13T11:57:50.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.494+0000 7fa0c2ffd700 1 -- 172.21.15.123:0/738316852 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa0c80095f0 con 0x7fa0d40f62a0 2024-01-13T11:57:50.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.494+0000 7fa0d359e700 1 -- 172.21.15.123:0/738316852 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0d4045ad0 con 0x7fa0d40f62a0 2024-01-13T11:57:50.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.494+0000 7fa0c2ffd700 1 --2- 172.21.15.123:0/738316852 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa0bc02ff60 0x7fa0bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:50.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.494+0000 7fa0c2ffd700 1 -- 172.21.15.123:0/738316852 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa0c8041000 con 0x7fa0d40f62a0 2024-01-13T11:57:50.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.495+0000 7fa0d1d9b700 1 --2- 172.21.15.123:0/738316852 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa0bc02ff60 0x7fa0bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:50.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.495+0000 7fa0d1d9b700 1 --2- 172.21.15.123:0/738316852 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa0bc02ff60 0x7fa0bc032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fa0c40096a0 tx=0x7fa0c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:50.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.499+0000 7fa0c2ffd700 1 -- 172.21.15.123:0/738316852 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa0c8014420 con 0x7fa0d40f62a0 2024-01-13T11:57:50.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.705+0000 7fa0d359e700 1 -- 172.21.15.123:0/738316852 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0d4107ee0 con 0x7fa0d40f62a0 2024-01-13T11:57:50.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.705+0000 7fa0c2ffd700 1 -- 172.21.15.123:0/738316852 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa0c8010230 con 0x7fa0d40f62a0 2024-01-13T11:57:50.693 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:50.693 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:50.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.707+0000 7fa0d359e700 1 -- 172.21.15.123:0/738316852 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa0bc02ff60 msgr2=0x7fa0bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:50.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.707+0000 7fa0d359e700 1 --2- 172.21.15.123:0/738316852 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa0bc02ff60 0x7fa0bc032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fa0c40096a0 tx=0x7fa0c4006b40).stop 2024-01-13T11:57:50.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.707+0000 7fa0d359e700 1 -- 172.21.15.123:0/738316852 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0d40f62a0 msgr2=0x7fa0d41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:50.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.707+0000 7fa0d359e700 1 --2- 172.21.15.123:0/738316852 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0d40f62a0 0x7fa0d41064b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fa0c8013510 tx=0x7fa0c800b040).stop 2024-01-13T11:57:50.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.707+0000 7fa0d359e700 1 -- 172.21.15.123:0/738316852 shutdown_connections 2024-01-13T11:57:50.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.707+0000 7fa0d359e700 1 --2- 172.21.15.123:0/738316852 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fa0bc02ff60 0x7fa0bc032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:50.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.707+0000 7fa0d359e700 1 --2- 172.21.15.123:0/738316852 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0d40f62a0 0x7fa0d41064b0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:50.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.707+0000 7fa0d359e700 1 -- 172.21.15.123:0/738316852 >> 172.21.15.123:0/738316852 conn(0x7fa0d40f1aa0 msgr2=0x7fa0d40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:50.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.707+0000 7fa0d359e700 1 -- 172.21.15.123:0/738316852 shutdown_connections 2024-01-13T11:57:50.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:50.707+0000 7fa0d359e700 1 -- 172.21.15.123:0/738316852 wait complete. 2024-01-13T11:57:50.695 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:51.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:49.578525+0000 mgr.smithi059.hmyldj (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:51.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:51 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:50.706598+0000 mon.smithi059 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.123:0/738316852' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:52.089 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:52.089 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:53.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.345+0000 7fb24a2f4700 1 -- 172.21.15.123:0/4061218903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2440f6a10 msgr2=0x7fb2440f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:53.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.345+0000 7fb24a2f4700 1 --2- 172.21.15.123:0/4061218903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2440f6a10 0x7fb2440f6e30 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fb238009fb0 tx=0x7fb238009b30).stop 2024-01-13T11:57:53.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.345+0000 7fb24a2f4700 1 -- 172.21.15.123:0/4061218903 shutdown_connections 2024-01-13T11:57:53.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.345+0000 7fb24a2f4700 1 --2- 172.21.15.123:0/4061218903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2440f6a10 0x7fb2440f6e30 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:53.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.345+0000 7fb24a2f4700 1 -- 172.21.15.123:0/4061218903 >> 172.21.15.123:0/4061218903 conn(0x7fb2440f02e0 msgr2=0x7fb2440f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:53.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.345+0000 7fb24a2f4700 1 -- 172.21.15.123:0/4061218903 shutdown_connections 2024-01-13T11:57:53.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.345+0000 7fb24a2f4700 1 -- 172.21.15.123:0/4061218903 wait complete. 2024-01-13T11:57:53.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.345+0000 7fb24a2f4700 1 Processor -- start 2024-01-13T11:57:53.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.346+0000 7fb24a2f4700 1 -- start start 2024-01-13T11:57:53.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.346+0000 7fb24a2f4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2440f6a10 0x7fb2441055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:53.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.346+0000 7fb24a2f4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb244105ac0 con 0x7fb2440f6a10 2024-01-13T11:57:53.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.346+0000 7fb243fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2440f6a10 0x7fb2441055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:53.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.346+0000 7fb243fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2440f6a10 0x7fb2441055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:35672/0 (socket says 172.21.15.123:35672) 2024-01-13T11:57:53.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.346+0000 7fb243fff700 1 -- 172.21.15.123:0/2945906395 learned_addr learned my addr 172.21.15.123:0/2945906395 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:53.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.346+0000 7fb243fff700 1 -- 172.21.15.123:0/2945906395 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb238005040 con 0x7fb2440f6a10 2024-01-13T11:57:53.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.346+0000 7fb243fff700 1 --2- 172.21.15.123:0/2945906395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2440f6a10 0x7fb2441055d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fb238014000 tx=0x7fb23800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:53.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.346+0000 7fb240ff9700 1 -- 172.21.15.123:0/2945906395 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2380095f0 con 0x7fb2440f6a10 2024-01-13T11:57:53.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.346+0000 7fb24a2f4700 1 -- 172.21.15.123:0/2945906395 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb244105cc0 con 0x7fb2440f6a10 2024-01-13T11:57:53.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.346+0000 7fb240ff9700 1 -- 172.21.15.123:0/2945906395 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb238007e70 con 0x7fb2440f6a10 2024-01-13T11:57:53.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.347+0000 7fb240ff9700 1 -- 172.21.15.123:0/2945906395 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb238014ea0 con 0x7fb2440f6a10 2024-01-13T11:57:53.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.347+0000 7fb24a2f4700 1 -- 172.21.15.123:0/2945906395 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb244106150 con 0x7fb2440f6a10 2024-01-13T11:57:53.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.347+0000 7fb240ff9700 1 -- 172.21.15.123:0/2945906395 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb238025070 con 0x7fb2440f6a10 2024-01-13T11:57:53.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.347+0000 7fb24a2f4700 1 -- 172.21.15.123:0/2945906395 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb244045ad0 con 0x7fb2440f6a10 2024-01-13T11:57:53.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.347+0000 7fb240ff9700 1 --2- 172.21.15.123:0/2945906395 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb22c02ff60 0x7fb22c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:53.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.348+0000 7fb240ff9700 1 -- 172.21.15.123:0/2945906395 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb238040930 con 0x7fb2440f6a10 2024-01-13T11:57:53.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.348+0000 7fb2437fe700 1 --2- 172.21.15.123:0/2945906395 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb22c02ff60 0x7fb22c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:53.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.348+0000 7fb2437fe700 1 --2- 172.21.15.123:0/2945906395 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb22c02ff60 0x7fb22c032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fb2340096a0 tx=0x7fb234006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:53.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.352+0000 7fb240ff9700 1 -- 172.21.15.123:0/2945906395 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb2380079c0 con 0x7fb2440f6a10 2024-01-13T11:57:53.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.555+0000 7fb24a2f4700 1 -- 172.21.15.123:0/2945906395 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb244106fa0 con 0x7fb2440f6a10 2024-01-13T11:57:53.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.556+0000 7fb240ff9700 1 -- 172.21.15.123:0/2945906395 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb23800f300 con 0x7fb2440f6a10 2024-01-13T11:57:53.543 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:53.543 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:53.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.557+0000 7fb24a2f4700 1 -- 172.21.15.123:0/2945906395 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb22c02ff60 msgr2=0x7fb22c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:53.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.557+0000 7fb24a2f4700 1 --2- 172.21.15.123:0/2945906395 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb22c02ff60 0x7fb22c032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fb2340096a0 tx=0x7fb234006b40).stop 2024-01-13T11:57:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.557+0000 7fb24a2f4700 1 -- 172.21.15.123:0/2945906395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2440f6a10 msgr2=0x7fb2441055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.557+0000 7fb24a2f4700 1 --2- 172.21.15.123:0/2945906395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2440f6a10 0x7fb2441055d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fb238014000 tx=0x7fb23800b040).stop 2024-01-13T11:57:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.557+0000 7fb24a2f4700 1 -- 172.21.15.123:0/2945906395 shutdown_connections 2024-01-13T11:57:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.557+0000 7fb24a2f4700 1 --2- 172.21.15.123:0/2945906395 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb22c02ff60 0x7fb22c032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.557+0000 7fb24a2f4700 1 --2- 172.21.15.123:0/2945906395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2440f6a10 0x7fb2441055d0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.557+0000 7fb24a2f4700 1 -- 172.21.15.123:0/2945906395 >> 172.21.15.123:0/2945906395 conn(0x7fb2440f02e0 msgr2=0x7fb2440f0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.557+0000 7fb24a2f4700 1 -- 172.21.15.123:0/2945906395 shutdown_connections 2024-01-13T11:57:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:53.557+0000 7fb24a2f4700 1 -- 172.21.15.123:0/2945906395 wait complete. 2024-01-13T11:57:53.545 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:53.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:53 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:51.578786+0000 mgr.smithi059.hmyldj (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:54.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:54 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:53.556874+0000 mon.smithi059 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.123:0/2945906395' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:54.974 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:54.974 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:55.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:55 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:53.579130+0000 mgr.smithi059.hmyldj (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:56.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.142+0000 7fc542d83700 1 -- 172.21.15.123:0/1521343986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 msgr2=0x7fc53c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:56.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.142+0000 7fc542d83700 1 --2- 172.21.15.123:0/1521343986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 0x7fc53c0f5830 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fc530009fb0 tx=0x7fc530009b30).stop 2024-01-13T11:57:56.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.142+0000 7fc542d83700 1 -- 172.21.15.123:0/1521343986 shutdown_connections 2024-01-13T11:57:56.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.142+0000 7fc542d83700 1 --2- 172.21.15.123:0/1521343986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 0x7fc53c0f5830 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:56.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.142+0000 7fc542d83700 1 -- 172.21.15.123:0/1521343986 >> 172.21.15.123:0/1521343986 conn(0x7fc53c0f0b70 msgr2=0x7fc53c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:56.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.142+0000 7fc542d83700 1 -- 172.21.15.123:0/1521343986 shutdown_connections 2024-01-13T11:57:56.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.142+0000 7fc542d83700 1 -- 172.21.15.123:0/1521343986 wait complete. 2024-01-13T11:57:56.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.142+0000 7fc542d83700 1 Processor -- start 2024-01-13T11:57:56.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.143+0000 7fc542d83700 1 -- start start 2024-01-13T11:57:56.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.143+0000 7fc542d83700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 0x7fc53c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:56.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.143+0000 7fc542d83700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c0fb820 con 0x7fc53c0f5410 2024-01-13T11:57:56.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.143+0000 7fc540b1f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 0x7fc53c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:56.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.143+0000 7fc540b1f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 0x7fc53c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:35696/0 (socket says 172.21.15.123:35696) 2024-01-13T11:57:56.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.143+0000 7fc540b1f700 1 -- 172.21.15.123:0/2480242466 learned_addr learned my addr 172.21.15.123:0/2480242466 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:56.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.143+0000 7fc540b1f700 1 -- 172.21.15.123:0/2480242466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc530005040 con 0x7fc53c0f5410 2024-01-13T11:57:56.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.143+0000 7fc540b1f700 1 --2- 172.21.15.123:0/2480242466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 0x7fc53c0fd3a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fc530013510 tx=0x7fc53000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:56.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.144+0000 7fc5397fa700 1 -- 172.21.15.123:0/2480242466 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc530009320 con 0x7fc53c0f5410 2024-01-13T11:57:56.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.144+0000 7fc542d83700 1 -- 172.21.15.123:0/2480242466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc53c0fb9c0 con 0x7fc53c0f5410 2024-01-13T11:57:56.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.144+0000 7fc5397fa700 1 -- 172.21.15.123:0/2480242466 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc530013f00 con 0x7fc53c0f5410 2024-01-13T11:57:56.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.144+0000 7fc5397fa700 1 -- 172.21.15.123:0/2480242466 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc530014d90 con 0x7fc53c0f5410 2024-01-13T11:57:56.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.144+0000 7fc542d83700 1 -- 172.21.15.123:0/2480242466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc53c0fbe50 con 0x7fc53c0f5410 2024-01-13T11:57:56.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.144+0000 7fc5397fa700 1 -- 172.21.15.123:0/2480242466 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5300095f0 con 0x7fc53c0f5410 2024-01-13T11:57:56.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.144+0000 7fc542d83700 1 -- 172.21.15.123:0/2480242466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc53c059100 con 0x7fc53c0f5410 2024-01-13T11:57:56.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.145+0000 7fc5397fa700 1 --2- 172.21.15.123:0/2480242466 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc52c02ff60 0x7fc52c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:56.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.145+0000 7fc5397fa700 1 -- 172.21.15.123:0/2480242466 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc5300410c0 con 0x7fc53c0f5410 2024-01-13T11:57:56.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.145+0000 7fc53bfff700 1 --2- 172.21.15.123:0/2480242466 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc52c02ff60 0x7fc52c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:56.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.145+0000 7fc53bfff700 1 --2- 172.21.15.123:0/2480242466 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc52c02ff60 0x7fc52c032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fc5280096a0 tx=0x7fc528006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:56.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.149+0000 7fc5397fa700 1 -- 172.21.15.123:0/2480242466 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc530014420 con 0x7fc53c0f5410 2024-01-13T11:57:56.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.353+0000 7fc542d83700 1 -- 172.21.15.123:0/2480242466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc53c0fae60 con 0x7fc53c0f5410 2024-01-13T11:57:56.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.353+0000 7fc5397fa700 1 -- 172.21.15.123:0/2480242466 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc530010230 con 0x7fc53c0f5410 2024-01-13T11:57:56.341 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:56.341 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:56.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.355+0000 7fc542d83700 1 -- 172.21.15.123:0/2480242466 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc52c02ff60 msgr2=0x7fc52c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:56.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.355+0000 7fc542d83700 1 --2- 172.21.15.123:0/2480242466 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc52c02ff60 0x7fc52c032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fc5280096a0 tx=0x7fc528006b40).stop 2024-01-13T11:57:56.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.355+0000 7fc542d83700 1 -- 172.21.15.123:0/2480242466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 msgr2=0x7fc53c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:56.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.355+0000 7fc542d83700 1 --2- 172.21.15.123:0/2480242466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 0x7fc53c0fd3a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fc530013510 tx=0x7fc53000b040).stop 2024-01-13T11:57:56.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.355+0000 7fc542d83700 1 -- 172.21.15.123:0/2480242466 shutdown_connections 2024-01-13T11:57:56.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.355+0000 7fc542d83700 1 --2- 172.21.15.123:0/2480242466 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc52c02ff60 0x7fc52c032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:56.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.355+0000 7fc542d83700 1 --2- 172.21.15.123:0/2480242466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc53c0f5410 0x7fc53c0fd3a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:56.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.355+0000 7fc542d83700 1 -- 172.21.15.123:0/2480242466 >> 172.21.15.123:0/2480242466 conn(0x7fc53c0f0b70 msgr2=0x7fc53c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:56.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.355+0000 7fc542d83700 1 -- 172.21.15.123:0/2480242466 shutdown_connections 2024-01-13T11:57:56.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:56.355+0000 7fc542d83700 1 -- 172.21.15.123:0/2480242466 wait complete. 2024-01-13T11:57:56.343 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:57.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:55.579461+0000 mgr.smithi059.hmyldj (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:57.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:57 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:56.354803+0000 mon.smithi059 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.123:0/2480242466' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:57:57.770 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:57:57.770 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:57:58.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.000+0000 7f0963ae1700 1 -- 172.21.15.123:0/2856405372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f095c0f5410 msgr2=0x7f095c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:58.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.000+0000 7f0963ae1700 1 --2- 172.21.15.123:0/2856405372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f095c0f5410 0x7f095c0f5830 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f0950009fb0 tx=0x7f0950009b30).stop 2024-01-13T11:57:58.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.000+0000 7f0963ae1700 1 -- 172.21.15.123:0/2856405372 shutdown_connections 2024-01-13T11:57:58.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.000+0000 7f0963ae1700 1 --2- 172.21.15.123:0/2856405372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f095c0f5410 0x7f095c0f5830 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:58.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.000+0000 7f0963ae1700 1 -- 172.21.15.123:0/2856405372 >> 172.21.15.123:0/2856405372 conn(0x7f095c0f0b70 msgr2=0x7f095c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:58.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.000+0000 7f0963ae1700 1 -- 172.21.15.123:0/2856405372 shutdown_connections 2024-01-13T11:57:58.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.000+0000 7f0963ae1700 1 -- 172.21.15.123:0/2856405372 wait complete. 2024-01-13T11:57:58.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.001+0000 7f0963ae1700 1 Processor -- start 2024-01-13T11:57:58.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.001+0000 7f0963ae1700 1 -- start start 2024-01-13T11:57:58.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.001+0000 7f0963ae1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f095c0f5410 0x7f095c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:58.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.001+0000 7f0963ae1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c105b20 con 0x7f095c0f5410 2024-01-13T11:57:58.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.001+0000 7f096187d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f095c0f5410 0x7f095c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:58.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.001+0000 7f096187d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f095c0f5410 0x7f095c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:42440/0 (socket says 172.21.15.123:42440) 2024-01-13T11:57:58.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.001+0000 7f096187d700 1 -- 172.21.15.123:0/1924256612 learned_addr learned my addr 172.21.15.123:0/1924256612 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:57:58.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.002+0000 7f096187d700 1 -- 172.21.15.123:0/1924256612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0950005040 con 0x7f095c0f5410 2024-01-13T11:57:58.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.002+0000 7f096187d700 1 --2- 172.21.15.123:0/1924256612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f095c0f5410 0x7f095c105630 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f0950013510 tx=0x7f095000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:58.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.002+0000 7f094e7fc700 1 -- 172.21.15.123:0/1924256612 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0950009320 con 0x7f095c0f5410 2024-01-13T11:57:58.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.002+0000 7f0963ae1700 1 -- 172.21.15.123:0/1924256612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095c105d20 con 0x7f095c0f5410 2024-01-13T11:57:58.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.002+0000 7f094e7fc700 1 -- 172.21.15.123:0/1924256612 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0950013f00 con 0x7f095c0f5410 2024-01-13T11:57:58.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.002+0000 7f094e7fc700 1 -- 172.21.15.123:0/1924256612 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0950014d90 con 0x7f095c0f5410 2024-01-13T11:57:58.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.002+0000 7f0963ae1700 1 -- 172.21.15.123:0/1924256612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f095c1061b0 con 0x7f095c0f5410 2024-01-13T11:57:58.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.003+0000 7f094e7fc700 1 -- 172.21.15.123:0/1924256612 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09500095f0 con 0x7f095c0f5410 2024-01-13T11:57:58.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.003+0000 7f0963ae1700 1 -- 172.21.15.123:0/1924256612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f095c045ad0 con 0x7f095c0f5410 2024-01-13T11:57:58.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.003+0000 7f094e7fc700 1 --2- 172.21.15.123:0/1924256612 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f094802ff60 0x7f0948032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:57:58.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.003+0000 7f094e7fc700 1 -- 172.21.15.123:0/1924256612 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0950040fe0 con 0x7f095c0f5410 2024-01-13T11:57:58.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.003+0000 7f096107c700 1 --2- 172.21.15.123:0/1924256612 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f094802ff60 0x7f0948032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:57:58.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.004+0000 7f096107c700 1 --2- 172.21.15.123:0/1924256612 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f094802ff60 0x7f0948032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f09580096a0 tx=0x7f0958006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:57:58.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.008+0000 7f094e7fc700 1 -- 172.21.15.123:0/1924256612 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f095001d7f0 con 0x7f095c0f5410 2024-01-13T11:57:59.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.211+0000 7f0963ae1700 1 -- 172.21.15.123:0/1924256612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f095c0fae60 con 0x7f095c0f5410 2024-01-13T11:57:59.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.212+0000 7f094e7fc700 1 -- 172.21.15.123:0/1924256612 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0950010230 con 0x7f095c0f5410 2024-01-13T11:57:59.199 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:57:59.200 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:57:59.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.214+0000 7f0963ae1700 1 -- 172.21.15.123:0/1924256612 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f094802ff60 msgr2=0x7f0948032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:59.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.214+0000 7f0963ae1700 1 --2- 172.21.15.123:0/1924256612 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f094802ff60 0x7f0948032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f09580096a0 tx=0x7f0958006b40).stop 2024-01-13T11:57:59.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.214+0000 7f0963ae1700 1 -- 172.21.15.123:0/1924256612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f095c0f5410 msgr2=0x7f095c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:57:59.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.214+0000 7f0963ae1700 1 --2- 172.21.15.123:0/1924256612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f095c0f5410 0x7f095c105630 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f0950013510 tx=0x7f095000b040).stop 2024-01-13T11:57:59.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.214+0000 7f0963ae1700 1 -- 172.21.15.123:0/1924256612 shutdown_connections 2024-01-13T11:57:59.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.214+0000 7f0963ae1700 1 --2- 172.21.15.123:0/1924256612 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f094802ff60 0x7f0948032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:59.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.214+0000 7f0963ae1700 1 --2- 172.21.15.123:0/1924256612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f095c0f5410 0x7f095c105630 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:57:59.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.214+0000 7f0963ae1700 1 -- 172.21.15.123:0/1924256612 >> 172.21.15.123:0/1924256612 conn(0x7f095c0f0b70 msgr2=0x7f095c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:57:59.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.214+0000 7f0963ae1700 1 -- 172.21.15.123:0/1924256612 shutdown_connections 2024-01-13T11:57:59.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:57:59.214+0000 7f0963ae1700 1 -- 172.21.15.123:0/1924256612 wait complete. 2024-01-13T11:57:59.202 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:57:59.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:59 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:57.579773+0000 mgr.smithi059.hmyldj (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:59.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:57:59 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:57:59.213529+0000 mon.smithi059 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.123:0/1924256612' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:58:00.577 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:58:00.578 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:58:01.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:01 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:57:59.580039+0000 mgr.smithi059.hmyldj (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:01.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.786+0000 7fcf6528f700 1 -- 172.21.15.123:0/1368994113 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf600f5410 msgr2=0x7fcf600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:01.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.786+0000 7fcf6528f700 1 --2- 172.21.15.123:0/1368994113 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf600f5410 0x7fcf600f5830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fcf54009fb0 tx=0x7fcf54009b30).stop 2024-01-13T11:58:01.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.786+0000 7fcf6528f700 1 -- 172.21.15.123:0/1368994113 shutdown_connections 2024-01-13T11:58:01.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.786+0000 7fcf6528f700 1 --2- 172.21.15.123:0/1368994113 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf600f5410 0x7fcf600f5830 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:01.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.786+0000 7fcf6528f700 1 -- 172.21.15.123:0/1368994113 >> 172.21.15.123:0/1368994113 conn(0x7fcf600f0b70 msgr2=0x7fcf600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:01.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.786+0000 7fcf6528f700 1 -- 172.21.15.123:0/1368994113 shutdown_connections 2024-01-13T11:58:01.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.786+0000 7fcf6528f700 1 -- 172.21.15.123:0/1368994113 wait complete. 2024-01-13T11:58:01.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.787+0000 7fcf6528f700 1 Processor -- start 2024-01-13T11:58:01.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.787+0000 7fcf6528f700 1 -- start start 2024-01-13T11:58:01.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.787+0000 7fcf6528f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf600f5410 0x7fcf60105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:01.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.787+0000 7fcf6528f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf60105b20 con 0x7fcf600f5410 2024-01-13T11:58:01.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.787+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf600f5410 0x7fcf60105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:01.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.787+0000 7fcf5effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf600f5410 0x7fcf60105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:42456/0 (socket says 172.21.15.123:42456) 2024-01-13T11:58:01.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.787+0000 7fcf5effd700 1 -- 172.21.15.123:0/125503491 learned_addr learned my addr 172.21.15.123:0/125503491 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:58:01.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.788+0000 7fcf5effd700 1 -- 172.21.15.123:0/125503491 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf54005040 con 0x7fcf600f5410 2024-01-13T11:58:01.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.788+0000 7fcf5effd700 1 --2- 172.21.15.123:0/125503491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf600f5410 0x7fcf60105630 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fcf54013510 tx=0x7fcf5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:01.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.788+0000 7fcf477fe700 1 -- 172.21.15.123:0/125503491 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf54009320 con 0x7fcf600f5410 2024-01-13T11:58:01.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.788+0000 7fcf477fe700 1 -- 172.21.15.123:0/125503491 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf54013f00 con 0x7fcf600f5410 2024-01-13T11:58:01.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.788+0000 7fcf6528f700 1 -- 172.21.15.123:0/125503491 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf60105d20 con 0x7fcf600f5410 2024-01-13T11:58:01.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.788+0000 7fcf477fe700 1 -- 172.21.15.123:0/125503491 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf54014d90 con 0x7fcf600f5410 2024-01-13T11:58:01.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.788+0000 7fcf6528f700 1 -- 172.21.15.123:0/125503491 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf601061b0 con 0x7fcf600f5410 2024-01-13T11:58:01.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.789+0000 7fcf477fe700 1 -- 172.21.15.123:0/125503491 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcf540263a0 con 0x7fcf600f5410 2024-01-13T11:58:01.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.789+0000 7fcf477fe700 1 --2- 172.21.15.123:0/125503491 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fcf4802ff60 0x7fcf48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:01.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.789+0000 7fcf477fe700 1 -- 172.21.15.123:0/125503491 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcf54041000 con 0x7fcf600f5410 2024-01-13T11:58:01.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.789+0000 7fcf5e7fc700 1 --2- 172.21.15.123:0/125503491 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fcf4802ff60 0x7fcf48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:01.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.790+0000 7fcf6528f700 1 -- 172.21.15.123:0/125503491 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf60059100 con 0x7fcf600f5410 2024-01-13T11:58:01.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.790+0000 7fcf5e7fc700 1 --2- 172.21.15.123:0/125503491 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fcf4802ff60 0x7fcf48032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fcf500096a0 tx=0x7fcf50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:01.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:01.795+0000 7fcf477fe700 1 -- 172.21.15.123:0/125503491 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf54012900 con 0x7fcf600f5410 2024-01-13T11:58:01.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:02.001+0000 7fcf6528f700 1 -- 172.21.15.123:0/125503491 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcf600fae60 con 0x7fcf600f5410 2024-01-13T11:58:01.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:02.001+0000 7fcf477fe700 1 -- 172.21.15.123:0/125503491 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcf5402b020 con 0x7fcf600f5410 2024-01-13T11:58:01.989 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:58:01.989 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:58:01.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:02.003+0000 7fcf6528f700 1 -- 172.21.15.123:0/125503491 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fcf4802ff60 msgr2=0x7fcf48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:01.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:02.003+0000 7fcf6528f700 1 --2- 172.21.15.123:0/125503491 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fcf4802ff60 0x7fcf48032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fcf500096a0 tx=0x7fcf50006b40).stop 2024-01-13T11:58:01.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:02.003+0000 7fcf6528f700 1 -- 172.21.15.123:0/125503491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf600f5410 msgr2=0x7fcf60105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:01.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:02.003+0000 7fcf6528f700 1 --2- 172.21.15.123:0/125503491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf600f5410 0x7fcf60105630 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fcf54013510 tx=0x7fcf5400b040).stop 2024-01-13T11:58:01.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:02.003+0000 7fcf6528f700 1 -- 172.21.15.123:0/125503491 shutdown_connections 2024-01-13T11:58:01.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:02.003+0000 7fcf6528f700 1 --2- 172.21.15.123:0/125503491 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fcf4802ff60 0x7fcf48032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:01.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:02.003+0000 7fcf6528f700 1 --2- 172.21.15.123:0/125503491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf600f5410 0x7fcf60105630 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:01.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:02.003+0000 7fcf6528f700 1 -- 172.21.15.123:0/125503491 >> 172.21.15.123:0/125503491 conn(0x7fcf600f0b70 msgr2=0x7fcf600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:01.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:02.003+0000 7fcf6528f700 1 -- 172.21.15.123:0/125503491 shutdown_connections 2024-01-13T11:58:01.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:02.003+0000 7fcf6528f700 1 -- 172.21.15.123:0/125503491 wait complete. 2024-01-13T11:58:01.991 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:58:02.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:02 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:58:02.002788+0000 mon.smithi059 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.123:0/125503491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:58:03.386 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:58:03.386 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:58:03.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:03 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:01.580412+0000 mgr.smithi059.hmyldj (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:04.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.580+0000 7f3143f51700 1 -- 172.21.15.123:0/932366410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f313c0f5410 msgr2=0x7f313c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:04.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.581+0000 7f3143f51700 1 --2- 172.21.15.123:0/932366410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f313c0f5410 0x7f313c0f5830 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f312c009fb0 tx=0x7f312c009b30).stop 2024-01-13T11:58:04.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.581+0000 7f3143f51700 1 -- 172.21.15.123:0/932366410 shutdown_connections 2024-01-13T11:58:04.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.581+0000 7f3143f51700 1 --2- 172.21.15.123:0/932366410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f313c0f5410 0x7f313c0f5830 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:04.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.581+0000 7f3143f51700 1 -- 172.21.15.123:0/932366410 >> 172.21.15.123:0/932366410 conn(0x7f313c0f0b70 msgr2=0x7f313c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:04.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.581+0000 7f3143f51700 1 -- 172.21.15.123:0/932366410 shutdown_connections 2024-01-13T11:58:04.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.581+0000 7f3143f51700 1 -- 172.21.15.123:0/932366410 wait complete. 2024-01-13T11:58:04.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.581+0000 7f3143f51700 1 Processor -- start 2024-01-13T11:58:04.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.581+0000 7f3143f51700 1 -- start start 2024-01-13T11:58:04.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.582+0000 7f3143f51700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f313c0f5410 0x7f313c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:04.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.582+0000 7f3143f51700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313c105b20 con 0x7f313c0f5410 2024-01-13T11:58:04.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.582+0000 7f3141ced700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f313c0f5410 0x7f313c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:04.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.582+0000 7f3141ced700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f313c0f5410 0x7f313c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:42482/0 (socket says 172.21.15.123:42482) 2024-01-13T11:58:04.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.582+0000 7f3141ced700 1 -- 172.21.15.123:0/2406611498 learned_addr learned my addr 172.21.15.123:0/2406611498 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:58:04.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.582+0000 7f3141ced700 1 -- 172.21.15.123:0/2406611498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c005040 con 0x7f313c0f5410 2024-01-13T11:58:04.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.582+0000 7f3141ced700 1 --2- 172.21.15.123:0/2406611498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f313c0f5410 0x7f313c105630 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f312c013510 tx=0x7f312c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:04.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.582+0000 7f31327fc700 1 -- 172.21.15.123:0/2406611498 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f312c009320 con 0x7f313c0f5410 2024-01-13T11:58:04.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.582+0000 7f3143f51700 1 -- 172.21.15.123:0/2406611498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313c105d20 con 0x7f313c0f5410 2024-01-13T11:58:04.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.583+0000 7f31327fc700 1 -- 172.21.15.123:0/2406611498 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f312c013f00 con 0x7f313c0f5410 2024-01-13T11:58:04.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.583+0000 7f31327fc700 1 -- 172.21.15.123:0/2406611498 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f312c014d90 con 0x7f313c0f5410 2024-01-13T11:58:04.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.583+0000 7f3143f51700 1 -- 172.21.15.123:0/2406611498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313c1061b0 con 0x7f313c0f5410 2024-01-13T11:58:04.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.583+0000 7f31327fc700 1 -- 172.21.15.123:0/2406611498 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f312c0095f0 con 0x7f313c0f5410 2024-01-13T11:58:04.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.583+0000 7f3143f51700 1 -- 172.21.15.123:0/2406611498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f313c045ad0 con 0x7f313c0f5410 2024-01-13T11:58:04.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.583+0000 7f31327fc700 1 --2- 172.21.15.123:0/2406611498 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f312802ff60 0x7f3128032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:04.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.584+0000 7f31327fc700 1 -- 172.21.15.123:0/2406611498 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f312c040fc0 con 0x7f313c0f5410 2024-01-13T11:58:04.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.584+0000 7f31414ec700 1 --2- 172.21.15.123:0/2406611498 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f312802ff60 0x7f3128032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:04.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.584+0000 7f31414ec700 1 --2- 172.21.15.123:0/2406611498 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f312802ff60 0x7f3128032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f31380096a0 tx=0x7f3138006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:04.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.588+0000 7f31327fc700 1 -- 172.21.15.123:0/2406611498 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f312c014420 con 0x7f313c0f5410 2024-01-13T11:58:04.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.795+0000 7f3143f51700 1 -- 172.21.15.123:0/2406611498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f313c0fae60 con 0x7f313c0f5410 2024-01-13T11:58:04.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.796+0000 7f31327fc700 1 -- 172.21.15.123:0/2406611498 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f312c010230 con 0x7f313c0f5410 2024-01-13T11:58:04.783 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:58:04.783 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:58:04.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.797+0000 7f3143f51700 1 -- 172.21.15.123:0/2406611498 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f312802ff60 msgr2=0x7f3128032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:04.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.797+0000 7f3143f51700 1 --2- 172.21.15.123:0/2406611498 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f312802ff60 0x7f3128032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f31380096a0 tx=0x7f3138006b40).stop 2024-01-13T11:58:04.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.797+0000 7f3143f51700 1 -- 172.21.15.123:0/2406611498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f313c0f5410 msgr2=0x7f313c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:04.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.797+0000 7f3143f51700 1 --2- 172.21.15.123:0/2406611498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f313c0f5410 0x7f313c105630 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f312c013510 tx=0x7f312c00b040).stop 2024-01-13T11:58:04.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.797+0000 7f3143f51700 1 -- 172.21.15.123:0/2406611498 shutdown_connections 2024-01-13T11:58:04.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.797+0000 7f3143f51700 1 --2- 172.21.15.123:0/2406611498 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f312802ff60 0x7f3128032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:04.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.797+0000 7f3143f51700 1 --2- 172.21.15.123:0/2406611498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f313c0f5410 0x7f313c105630 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:04.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.797+0000 7f3143f51700 1 -- 172.21.15.123:0/2406611498 >> 172.21.15.123:0/2406611498 conn(0x7f313c0f0b70 msgr2=0x7f313c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:04.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.797+0000 7f3143f51700 1 -- 172.21.15.123:0/2406611498 shutdown_connections 2024-01-13T11:58:04.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:04.797+0000 7f3143f51700 1 -- 172.21.15.123:0/2406611498 wait complete. 2024-01-13T11:58:04.786 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:58:05.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:05 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:03.580804+0000 mgr.smithi059.hmyldj (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:05.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:05 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:58:04.797172+0000 mon.smithi059 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.123:0/2406611498' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:58:06.179 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:58:06.180 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:58:07.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.392+0000 7fce50806700 1 -- 172.21.15.123:0/3604195456 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce480138a0 con 0x7fce4c0f51a0 2024-01-13T11:58:07.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.393+0000 7fce5280a700 1 -- 172.21.15.123:0/3604195456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce4c0f51a0 msgr2=0x7fce4c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:07.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.393+0000 7fce5280a700 1 --2- 172.21.15.123:0/3604195456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce4c0f51a0 0x7fce4c0f55c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fce48009fb0 tx=0x7fce48009b30).stop 2024-01-13T11:58:07.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.393+0000 7fce5280a700 1 -- 172.21.15.123:0/3604195456 shutdown_connections 2024-01-13T11:58:07.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.393+0000 7fce5280a700 1 --2- 172.21.15.123:0/3604195456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce4c0f51a0 0x7fce4c0f55c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:07.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.393+0000 7fce5280a700 1 -- 172.21.15.123:0/3604195456 >> 172.21.15.123:0/3604195456 conn(0x7fce4c0f09a0 msgr2=0x7fce4c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:07.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.393+0000 7fce5280a700 1 -- 172.21.15.123:0/3604195456 shutdown_connections 2024-01-13T11:58:07.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.393+0000 7fce5280a700 1 -- 172.21.15.123:0/3604195456 wait complete. 2024-01-13T11:58:07.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.393+0000 7fce5280a700 1 Processor -- start 2024-01-13T11:58:07.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.394+0000 7fce5280a700 1 -- start start 2024-01-13T11:58:07.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.394+0000 7fce5280a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce4c0f51a0 0x7fce4c1086b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:07.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.394+0000 7fce5280a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4c108ba0 con 0x7fce4c0f51a0 2024-01-13T11:58:07.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.394+0000 7fce51808700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce4c0f51a0 0x7fce4c1086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:07.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.394+0000 7fce51808700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce4c0f51a0 0x7fce4c1086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:35714/0 (socket says 172.21.15.123:35714) 2024-01-13T11:58:07.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.394+0000 7fce51808700 1 -- 172.21.15.123:0/1064115621 learned_addr learned my addr 172.21.15.123:0/1064115621 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:58:07.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.394+0000 7fce51808700 1 -- 172.21.15.123:0/1064115621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce48005040 con 0x7fce4c0f51a0 2024-01-13T11:58:07.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.394+0000 7fce51808700 1 --2- 172.21.15.123:0/1064115621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce4c0f51a0 0x7fce4c1086b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fce48013510 tx=0x7fce4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:07.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.394+0000 7fce427fc700 1 -- 172.21.15.123:0/1064115621 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce48009320 con 0x7fce4c0f51a0 2024-01-13T11:58:07.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.394+0000 7fce5280a700 1 -- 172.21.15.123:0/1064115621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce4c108da0 con 0x7fce4c0f51a0 2024-01-13T11:58:07.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.395+0000 7fce427fc700 1 -- 172.21.15.123:0/1064115621 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce48013f00 con 0x7fce4c0f51a0 2024-01-13T11:58:07.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.395+0000 7fce427fc700 1 -- 172.21.15.123:0/1064115621 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce48014dc0 con 0x7fce4c0f51a0 2024-01-13T11:58:07.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.395+0000 7fce5280a700 1 -- 172.21.15.123:0/1064115621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce4c109230 con 0x7fce4c0f51a0 2024-01-13T11:58:07.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.395+0000 7fce427fc700 1 -- 172.21.15.123:0/1064115621 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce480095f0 con 0x7fce4c0f51a0 2024-01-13T11:58:07.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.395+0000 7fce5280a700 1 -- 172.21.15.123:0/1064115621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce4c045ad0 con 0x7fce4c0f51a0 2024-01-13T11:58:07.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.395+0000 7fce427fc700 1 --2- 172.21.15.123:0/1064115621 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce3802ff60 0x7fce38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:07.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.396+0000 7fce427fc700 1 -- 172.21.15.123:0/1064115621 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce48041000 con 0x7fce4c0f51a0 2024-01-13T11:58:07.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.396+0000 7fce51007700 1 --2- 172.21.15.123:0/1064115621 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce3802ff60 0x7fce38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:07.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.396+0000 7fce51007700 1 --2- 172.21.15.123:0/1064115621 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce3802ff60 0x7fce38032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fce3c0096a0 tx=0x7fce3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:07.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.401+0000 7fce427fc700 1 -- 172.21.15.123:0/1064115621 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce48014420 con 0x7fce4c0f51a0 2024-01-13T11:58:07.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.605+0000 7fce5280a700 1 -- 172.21.15.123:0/1064115621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce4c10a0e0 con 0x7fce4c0f51a0 2024-01-13T11:58:07.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.606+0000 7fce427fc700 1 -- 172.21.15.123:0/1064115621 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce48010230 con 0x7fce4c0f51a0 2024-01-13T11:58:07.593 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:58:07.593 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:58:07.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.607+0000 7fce5280a700 1 -- 172.21.15.123:0/1064115621 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce3802ff60 msgr2=0x7fce38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:07.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.607+0000 7fce5280a700 1 --2- 172.21.15.123:0/1064115621 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce3802ff60 0x7fce38032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fce3c0096a0 tx=0x7fce3c006b40).stop 2024-01-13T11:58:07.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.607+0000 7fce5280a700 1 -- 172.21.15.123:0/1064115621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce4c0f51a0 msgr2=0x7fce4c1086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:07.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.607+0000 7fce5280a700 1 --2- 172.21.15.123:0/1064115621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce4c0f51a0 0x7fce4c1086b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fce48013510 tx=0x7fce4800b040).stop 2024-01-13T11:58:07.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.607+0000 7fce5280a700 1 -- 172.21.15.123:0/1064115621 shutdown_connections 2024-01-13T11:58:07.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.607+0000 7fce5280a700 1 --2- 172.21.15.123:0/1064115621 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fce3802ff60 0x7fce38032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:07.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.607+0000 7fce5280a700 1 --2- 172.21.15.123:0/1064115621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce4c0f51a0 0x7fce4c1086b0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:07.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.607+0000 7fce5280a700 1 -- 172.21.15.123:0/1064115621 >> 172.21.15.123:0/1064115621 conn(0x7fce4c0f09a0 msgr2=0x7fce4c0f1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:07.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.607+0000 7fce5280a700 1 -- 172.21.15.123:0/1064115621 shutdown_connections 2024-01-13T11:58:07.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:07.607+0000 7fce5280a700 1 -- 172.21.15.123:0/1064115621 wait complete. 2024-01-13T11:58:07.596 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:58:07.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:07 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:05.581061+0000 mgr.smithi059.hmyldj (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:08.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:08 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:58:07.607040+0000 mon.smithi059 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.123:0/1064115621' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:58:08.967 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:58:08.967 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:58:09.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:09 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:07.581421+0000 mgr.smithi059.hmyldj (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:10.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.163+0000 7f859ce53700 1 -- 172.21.15.123:0/618325973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85980f5410 msgr2=0x7f85980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:10.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.163+0000 7f859ce53700 1 --2- 172.21.15.123:0/618325973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85980f5410 0x7f85980f5830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f858c009fb0 tx=0x7f858c009b30).stop 2024-01-13T11:58:10.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.163+0000 7f859ce53700 1 -- 172.21.15.123:0/618325973 shutdown_connections 2024-01-13T11:58:10.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.163+0000 7f859ce53700 1 --2- 172.21.15.123:0/618325973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85980f5410 0x7f85980f5830 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:10.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.163+0000 7f859ce53700 1 -- 172.21.15.123:0/618325973 >> 172.21.15.123:0/618325973 conn(0x7f85980f0b70 msgr2=0x7f85980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:10.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.163+0000 7f859ce53700 1 -- 172.21.15.123:0/618325973 shutdown_connections 2024-01-13T11:58:10.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.163+0000 7f859ce53700 1 -- 172.21.15.123:0/618325973 wait complete. 2024-01-13T11:58:10.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.163+0000 7f859ce53700 1 Processor -- start 2024-01-13T11:58:10.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.163+0000 7f859ce53700 1 -- start start 2024-01-13T11:58:10.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.163+0000 7f859ce53700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85980f5410 0x7f8598109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:10.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.163+0000 7f859ce53700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8598060240 con 0x7f85980f5410 2024-01-13T11:58:10.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.164+0000 7f859659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85980f5410 0x7f8598109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:10.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.164+0000 7f859659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85980f5410 0x7f8598109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:35722/0 (socket says 172.21.15.123:35722) 2024-01-13T11:58:10.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.164+0000 7f859659c700 1 -- 172.21.15.123:0/1537904500 learned_addr learned my addr 172.21.15.123:0/1537904500 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:58:10.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.164+0000 7f859659c700 1 -- 172.21.15.123:0/1537904500 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858c005040 con 0x7f85980f5410 2024-01-13T11:58:10.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.164+0000 7f859659c700 1 --2- 172.21.15.123:0/1537904500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85980f5410 0x7f8598109430 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f858c013510 tx=0x7f858c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:10.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.164+0000 7f8586ffd700 1 -- 172.21.15.123:0/1537904500 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f858c009320 con 0x7f85980f5410 2024-01-13T11:58:10.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.164+0000 7f859ce53700 1 -- 172.21.15.123:0/1537904500 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8598060440 con 0x7f85980f5410 2024-01-13T11:58:10.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.164+0000 7f8586ffd700 1 -- 172.21.15.123:0/1537904500 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f858c013f00 con 0x7f85980f5410 2024-01-13T11:58:10.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.164+0000 7f8586ffd700 1 -- 172.21.15.123:0/1537904500 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f858c014d90 con 0x7f85980f5410 2024-01-13T11:58:10.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.164+0000 7f859ce53700 1 -- 172.21.15.123:0/1537904500 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8598109ae0 con 0x7f85980f5410 2024-01-13T11:58:10.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.165+0000 7f8586ffd700 1 -- 172.21.15.123:0/1537904500 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f858c0095f0 con 0x7f85980f5410 2024-01-13T11:58:10.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.165+0000 7f859ce53700 1 -- 172.21.15.123:0/1537904500 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8598045ad0 con 0x7f85980f5410 2024-01-13T11:58:10.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.165+0000 7f8586ffd700 1 --2- 172.21.15.123:0/1537904500 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f858002ff60 0x7f8580032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:10.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.165+0000 7f8586ffd700 1 -- 172.21.15.123:0/1537904500 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f858c040fe0 con 0x7f85980f5410 2024-01-13T11:58:10.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.165+0000 7f8595d9b700 1 --2- 172.21.15.123:0/1537904500 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f858002ff60 0x7f8580032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:10.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.166+0000 7f8595d9b700 1 --2- 172.21.15.123:0/1537904500 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f858002ff60 0x7f8580032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f85880096a0 tx=0x7f8588006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:10.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.170+0000 7f8586ffd700 1 -- 172.21.15.123:0/1537904500 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f858c01d7f0 con 0x7f85980f5410 2024-01-13T11:58:10.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.376+0000 7f859ce53700 1 -- 172.21.15.123:0/1537904500 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85980fae60 con 0x7f85980f5410 2024-01-13T11:58:10.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.377+0000 7f8586ffd700 1 -- 172.21.15.123:0/1537904500 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f858c010230 con 0x7f85980f5410 2024-01-13T11:58:10.364 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:58:10.364 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:58:10.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.378+0000 7f859ce53700 1 -- 172.21.15.123:0/1537904500 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f858002ff60 msgr2=0x7f8580032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:10.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.378+0000 7f859ce53700 1 --2- 172.21.15.123:0/1537904500 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f858002ff60 0x7f8580032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f85880096a0 tx=0x7f8588006b40).stop 2024-01-13T11:58:10.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.378+0000 7f859ce53700 1 -- 172.21.15.123:0/1537904500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85980f5410 msgr2=0x7f8598109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:10.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.378+0000 7f859ce53700 1 --2- 172.21.15.123:0/1537904500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85980f5410 0x7f8598109430 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f858c013510 tx=0x7f858c00b040).stop 2024-01-13T11:58:10.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.378+0000 7f859ce53700 1 -- 172.21.15.123:0/1537904500 shutdown_connections 2024-01-13T11:58:10.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.378+0000 7f859ce53700 1 --2- 172.21.15.123:0/1537904500 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f858002ff60 0x7f8580032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:10.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.378+0000 7f859ce53700 1 --2- 172.21.15.123:0/1537904500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85980f5410 0x7f8598109430 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:10.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.378+0000 7f859ce53700 1 -- 172.21.15.123:0/1537904500 >> 172.21.15.123:0/1537904500 conn(0x7f85980f0b70 msgr2=0x7f85980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:10.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.378+0000 7f859ce53700 1 -- 172.21.15.123:0/1537904500 shutdown_connections 2024-01-13T11:58:10.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:10.378+0000 7f859ce53700 1 -- 172.21.15.123:0/1537904500 wait complete. 2024-01-13T11:58:10.366 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:58:11.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:11 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:09.581749+0000 mgr.smithi059.hmyldj (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:11.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:11 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:58:10.378068+0000 mon.smithi059 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.123:0/1537904500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:58:11.755 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:58:11.755 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:58:12.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.915+0000 7fcb9cfc9700 1 -- 172.21.15.123:0/135750769 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb980f5330 msgr2=0x7fcb980f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:12.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.915+0000 7fcb9cfc9700 1 --2- 172.21.15.123:0/135750769 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb980f5330 0x7fcb980f5750 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fcb88009fb0 tx=0x7fcb88009b30).stop 2024-01-13T11:58:12.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.915+0000 7fcb9cfc9700 1 -- 172.21.15.123:0/135750769 shutdown_connections 2024-01-13T11:58:12.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.915+0000 7fcb9cfc9700 1 --2- 172.21.15.123:0/135750769 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb980f5330 0x7fcb980f5750 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:12.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.915+0000 7fcb9cfc9700 1 -- 172.21.15.123:0/135750769 >> 172.21.15.123:0/135750769 conn(0x7fcb980f0a90 msgr2=0x7fcb980f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:12.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.915+0000 7fcb9cfc9700 1 -- 172.21.15.123:0/135750769 shutdown_connections 2024-01-13T11:58:12.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.915+0000 7fcb9cfc9700 1 -- 172.21.15.123:0/135750769 wait complete. 2024-01-13T11:58:12.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.916+0000 7fcb9cfc9700 1 Processor -- start 2024-01-13T11:58:12.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.916+0000 7fcb9cfc9700 1 -- start start 2024-01-13T11:58:12.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.916+0000 7fcb9cfc9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb980f5330 0x7fcb9810baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:12.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.916+0000 7fcb9cfc9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb981074e0 con 0x7fcb980f5330 2024-01-13T11:58:12.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.916+0000 7fcb9659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb980f5330 0x7fcb9810baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:12.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.916+0000 7fcb9659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb980f5330 0x7fcb9810baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:35740/0 (socket says 172.21.15.123:35740) 2024-01-13T11:58:12.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.916+0000 7fcb9659c700 1 -- 172.21.15.123:0/2131195645 learned_addr learned my addr 172.21.15.123:0/2131195645 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:58:12.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.917+0000 7fcb9659c700 1 -- 172.21.15.123:0/2131195645 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb88005040 con 0x7fcb980f5330 2024-01-13T11:58:12.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.917+0000 7fcb9659c700 1 --2- 172.21.15.123:0/2131195645 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb980f5330 0x7fcb9810baf0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fcb88013510 tx=0x7fcb8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:12.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.917+0000 7fcb86ffd700 1 -- 172.21.15.123:0/2131195645 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb88009320 con 0x7fcb980f5330 2024-01-13T11:58:12.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.917+0000 7fcb9cfc9700 1 -- 172.21.15.123:0/2131195645 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb98060220 con 0x7fcb980f5330 2024-01-13T11:58:12.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.917+0000 7fcb86ffd700 1 -- 172.21.15.123:0/2131195645 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb88013f00 con 0x7fcb980f5330 2024-01-13T11:58:12.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.917+0000 7fcb86ffd700 1 -- 172.21.15.123:0/2131195645 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb88014d90 con 0x7fcb980f5330 2024-01-13T11:58:12.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.917+0000 7fcb9cfc9700 1 -- 172.21.15.123:0/2131195645 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9810bfe0 con 0x7fcb980f5330 2024-01-13T11:58:12.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.918+0000 7fcb86ffd700 1 -- 172.21.15.123:0/2131195645 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb880095f0 con 0x7fcb980f5330 2024-01-13T11:58:12.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.918+0000 7fcb9cfc9700 1 -- 172.21.15.123:0/2131195645 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb98045ad0 con 0x7fcb980f5330 2024-01-13T11:58:12.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.918+0000 7fcb86ffd700 1 --2- 172.21.15.123:0/2131195645 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fcb8002ff60 0x7fcb80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:12.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.918+0000 7fcb86ffd700 1 -- 172.21.15.123:0/2131195645 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb88040fd0 con 0x7fcb980f5330 2024-01-13T11:58:12.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.918+0000 7fcb95d9b700 1 --2- 172.21.15.123:0/2131195645 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fcb8002ff60 0x7fcb80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:12.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.919+0000 7fcb95d9b700 1 --2- 172.21.15.123:0/2131195645 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fcb8002ff60 0x7fcb80032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fcb8c0096a0 tx=0x7fcb8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:12.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:12.923+0000 7fcb86ffd700 1 -- 172.21.15.123:0/2131195645 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb88014420 con 0x7fcb980f5330 2024-01-13T11:58:13.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:13.130+0000 7fcb9cfc9700 1 -- 172.21.15.123:0/2131195645 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb9805ea50 con 0x7fcb980f5330 2024-01-13T11:58:13.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:13.130+0000 7fcb86ffd700 1 -- 172.21.15.123:0/2131195645 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcb8801d7f0 con 0x7fcb980f5330 2024-01-13T11:58:13.116 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:58:13.117 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:58:13.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:13.132+0000 7fcb9cfc9700 1 -- 172.21.15.123:0/2131195645 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fcb8002ff60 msgr2=0x7fcb80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:13.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:13.132+0000 7fcb9cfc9700 1 --2- 172.21.15.123:0/2131195645 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fcb8002ff60 0x7fcb80032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fcb8c0096a0 tx=0x7fcb8c006b40).stop 2024-01-13T11:58:13.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:13.132+0000 7fcb9cfc9700 1 -- 172.21.15.123:0/2131195645 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb980f5330 msgr2=0x7fcb9810baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:13.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:13.132+0000 7fcb9cfc9700 1 --2- 172.21.15.123:0/2131195645 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb980f5330 0x7fcb9810baf0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fcb88013510 tx=0x7fcb8800b040).stop 2024-01-13T11:58:13.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:13.132+0000 7fcb9cfc9700 1 -- 172.21.15.123:0/2131195645 shutdown_connections 2024-01-13T11:58:13.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:13.132+0000 7fcb9cfc9700 1 --2- 172.21.15.123:0/2131195645 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fcb8002ff60 0x7fcb80032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:13.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:13.132+0000 7fcb9cfc9700 1 --2- 172.21.15.123:0/2131195645 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb980f5330 0x7fcb9810baf0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:13.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:13.132+0000 7fcb9cfc9700 1 -- 172.21.15.123:0/2131195645 >> 172.21.15.123:0/2131195645 conn(0x7fcb980f0a90 msgr2=0x7fcb980f1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:13.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:13.132+0000 7fcb9cfc9700 1 -- 172.21.15.123:0/2131195645 shutdown_connections 2024-01-13T11:58:13.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:13.132+0000 7fcb9cfc9700 1 -- 172.21.15.123:0/2131195645 wait complete. 2024-01-13T11:58:13.119 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:58:13.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:13 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:11.582069+0000 mgr.smithi059.hmyldj (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:13.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:13 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:58:13.130624+0000 mon.smithi059 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.123:0/2131195645' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:58:14.508 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:58:14.508 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:58:15.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:15 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:13.582506+0000 mgr.smithi059.hmyldj (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:15.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.663+0000 7f8ecf3c5700 1 -- 172.21.15.123:0/2693410744 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ec80f5410 msgr2=0x7f8ec80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:15.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.663+0000 7f8ecf3c5700 1 --2- 172.21.15.123:0/2693410744 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ec80f5410 0x7f8ec80f5830 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f8eb8009fb0 tx=0x7f8eb8009b30).stop 2024-01-13T11:58:15.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.663+0000 7f8ecf3c5700 1 -- 172.21.15.123:0/2693410744 shutdown_connections 2024-01-13T11:58:15.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.663+0000 7f8ecf3c5700 1 --2- 172.21.15.123:0/2693410744 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ec80f5410 0x7f8ec80f5830 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:15.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.663+0000 7f8ecf3c5700 1 -- 172.21.15.123:0/2693410744 >> 172.21.15.123:0/2693410744 conn(0x7f8ec80f0b70 msgr2=0x7f8ec80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:15.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.663+0000 7f8ecf3c5700 1 -- 172.21.15.123:0/2693410744 shutdown_connections 2024-01-13T11:58:15.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.663+0000 7f8ecf3c5700 1 -- 172.21.15.123:0/2693410744 wait complete. 2024-01-13T11:58:15.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.664+0000 7f8ecf3c5700 1 Processor -- start 2024-01-13T11:58:15.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.664+0000 7f8ecf3c5700 1 -- start start 2024-01-13T11:58:15.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.664+0000 7f8ecf3c5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ec80f5410 0x7f8ec8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:15.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.664+0000 7f8ecf3c5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec8060240 con 0x7f8ec80f5410 2024-01-13T11:58:15.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.664+0000 7f8ecd161700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ec80f5410 0x7f8ec8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:15.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.664+0000 7f8ecd161700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ec80f5410 0x7f8ec8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:35748/0 (socket says 172.21.15.123:35748) 2024-01-13T11:58:15.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.664+0000 7f8ecd161700 1 -- 172.21.15.123:0/3366749010 learned_addr learned my addr 172.21.15.123:0/3366749010 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:58:15.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.665+0000 7f8ecd161700 1 -- 172.21.15.123:0/3366749010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb8005040 con 0x7f8ec80f5410 2024-01-13T11:58:15.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.665+0000 7f8ecd161700 1 --2- 172.21.15.123:0/3366749010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ec80f5410 0x7f8ec8109430 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f8eb8013510 tx=0x7f8eb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:15.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.665+0000 7f8ebdffb700 1 -- 172.21.15.123:0/3366749010 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8eb8009320 con 0x7f8ec80f5410 2024-01-13T11:58:15.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.665+0000 7f8ecf3c5700 1 -- 172.21.15.123:0/3366749010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec8060440 con 0x7f8ec80f5410 2024-01-13T11:58:15.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.665+0000 7f8ebdffb700 1 -- 172.21.15.123:0/3366749010 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8eb8013f00 con 0x7f8ec80f5410 2024-01-13T11:58:15.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.665+0000 7f8ebdffb700 1 -- 172.21.15.123:0/3366749010 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8eb8014d90 con 0x7f8ec80f5410 2024-01-13T11:58:15.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.665+0000 7f8ecf3c5700 1 -- 172.21.15.123:0/3366749010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec8109ae0 con 0x7f8ec80f5410 2024-01-13T11:58:15.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.666+0000 7f8ebdffb700 1 -- 172.21.15.123:0/3366749010 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8eb80095f0 con 0x7f8ec80f5410 2024-01-13T11:58:15.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.666+0000 7f8ecf3c5700 1 -- 172.21.15.123:0/3366749010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec8045ad0 con 0x7f8ec80f5410 2024-01-13T11:58:15.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.666+0000 7f8ebdffb700 1 --2- 172.21.15.123:0/3366749010 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8eb402ff60 0x7f8eb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:15.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.666+0000 7f8ebdffb700 1 -- 172.21.15.123:0/3366749010 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8eb8040fc0 con 0x7f8ec80f5410 2024-01-13T11:58:15.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.666+0000 7f8ecc960700 1 --2- 172.21.15.123:0/3366749010 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8eb402ff60 0x7f8eb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:15.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.667+0000 7f8ecc960700 1 --2- 172.21.15.123:0/3366749010 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8eb402ff60 0x7f8eb4032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f8ec40096a0 tx=0x7f8ec4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:15.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.671+0000 7f8ebdffb700 1 -- 172.21.15.123:0/3366749010 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8eb8014420 con 0x7f8ec80f5410 2024-01-13T11:58:15.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.876+0000 7f8ecf3c5700 1 -- 172.21.15.123:0/3366749010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ec80fae60 con 0x7f8ec80f5410 2024-01-13T11:58:15.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.876+0000 7f8ebdffb700 1 -- 172.21.15.123:0/3366749010 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8eb8010230 con 0x7f8ec80f5410 2024-01-13T11:58:15.863 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:58:15.863 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:58:15.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.878+0000 7f8ecf3c5700 1 -- 172.21.15.123:0/3366749010 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8eb402ff60 msgr2=0x7f8eb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:15.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.878+0000 7f8ecf3c5700 1 --2- 172.21.15.123:0/3366749010 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8eb402ff60 0x7f8eb4032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f8ec40096a0 tx=0x7f8ec4006b40).stop 2024-01-13T11:58:15.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.878+0000 7f8ecf3c5700 1 -- 172.21.15.123:0/3366749010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ec80f5410 msgr2=0x7f8ec8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:15.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.878+0000 7f8ecf3c5700 1 --2- 172.21.15.123:0/3366749010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ec80f5410 0x7f8ec8109430 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f8eb8013510 tx=0x7f8eb800b040).stop 2024-01-13T11:58:15.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.878+0000 7f8ecf3c5700 1 -- 172.21.15.123:0/3366749010 shutdown_connections 2024-01-13T11:58:15.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.878+0000 7f8ecf3c5700 1 --2- 172.21.15.123:0/3366749010 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8eb402ff60 0x7f8eb4032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:15.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.878+0000 7f8ecf3c5700 1 --2- 172.21.15.123:0/3366749010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ec80f5410 0x7f8ec8109430 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:15.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.878+0000 7f8ecf3c5700 1 -- 172.21.15.123:0/3366749010 >> 172.21.15.123:0/3366749010 conn(0x7f8ec80f0b70 msgr2=0x7f8ec80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:15.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.878+0000 7f8ecf3c5700 1 -- 172.21.15.123:0/3366749010 shutdown_connections 2024-01-13T11:58:15.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:15.878+0000 7f8ecf3c5700 1 -- 172.21.15.123:0/3366749010 wait complete. 2024-01-13T11:58:15.866 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:58:16.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:16 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:58:15.876934+0000 mon.smithi059 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.123:0/3366749010' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:58:17.258 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:58:17.259 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:58:17.635 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:17 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:15.582841+0000 mgr.smithi059.hmyldj (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:18.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.452+0000 7fc3a5884700 1 -- 172.21.15.123:0/3583279956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3a00f5410 msgr2=0x7fc3a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:18.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.452+0000 7fc3a5884700 1 --2- 172.21.15.123:0/3583279956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3a00f5410 0x7fc3a00f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fc394009fb0 tx=0x7fc394009b30).stop 2024-01-13T11:58:18.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.452+0000 7fc3a5884700 1 -- 172.21.15.123:0/3583279956 shutdown_connections 2024-01-13T11:58:18.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.452+0000 7fc3a5884700 1 --2- 172.21.15.123:0/3583279956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3a00f5410 0x7fc3a00f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:18.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.452+0000 7fc3a5884700 1 -- 172.21.15.123:0/3583279956 >> 172.21.15.123:0/3583279956 conn(0x7fc3a00f0b70 msgr2=0x7fc3a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:18.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.452+0000 7fc3a5884700 1 -- 172.21.15.123:0/3583279956 shutdown_connections 2024-01-13T11:58:18.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.452+0000 7fc3a5884700 1 -- 172.21.15.123:0/3583279956 wait complete. 2024-01-13T11:58:18.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.453+0000 7fc3a5884700 1 Processor -- start 2024-01-13T11:58:18.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.453+0000 7fc3a5884700 1 -- start start 2024-01-13T11:58:18.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.453+0000 7fc3a5884700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3a00f5410 0x7fc3a0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:18.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.453+0000 7fc3a5884700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a0060220 con 0x7fc3a00f5410 2024-01-13T11:58:18.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.453+0000 7fc39effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3a00f5410 0x7fc3a0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:18.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.453+0000 7fc39effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3a00f5410 0x7fc3a0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:37680/0 (socket says 172.21.15.123:37680) 2024-01-13T11:58:18.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.453+0000 7fc39effd700 1 -- 172.21.15.123:0/1387195409 learned_addr learned my addr 172.21.15.123:0/1387195409 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:58:18.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.454+0000 7fc39effd700 1 -- 172.21.15.123:0/1387195409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc394005040 con 0x7fc3a00f5410 2024-01-13T11:58:18.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.454+0000 7fc39effd700 1 --2- 172.21.15.123:0/1387195409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3a00f5410 0x7fc3a0109450 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fc394013510 tx=0x7fc39400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:18.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.454+0000 7fc3877fe700 1 -- 172.21.15.123:0/1387195409 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc394009320 con 0x7fc3a00f5410 2024-01-13T11:58:18.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.454+0000 7fc3a5884700 1 -- 172.21.15.123:0/1387195409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a0060420 con 0x7fc3a00f5410 2024-01-13T11:58:18.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.454+0000 7fc3877fe700 1 -- 172.21.15.123:0/1387195409 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc394013f00 con 0x7fc3a00f5410 2024-01-13T11:58:18.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.454+0000 7fc3877fe700 1 -- 172.21.15.123:0/1387195409 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc394014d90 con 0x7fc3a00f5410 2024-01-13T11:58:18.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.454+0000 7fc3a5884700 1 -- 172.21.15.123:0/1387195409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a0109b00 con 0x7fc3a00f5410 2024-01-13T11:58:18.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.455+0000 7fc3877fe700 1 -- 172.21.15.123:0/1387195409 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc394014420 con 0x7fc3a00f5410 2024-01-13T11:58:18.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.455+0000 7fc3a5884700 1 -- 172.21.15.123:0/1387195409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a0059100 con 0x7fc3a00f5410 2024-01-13T11:58:18.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.455+0000 7fc3877fe700 1 --2- 172.21.15.123:0/1387195409 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc38802ff60 0x7fc388032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:18.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.455+0000 7fc3877fe700 1 -- 172.21.15.123:0/1387195409 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc394041050 con 0x7fc3a00f5410 2024-01-13T11:58:18.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.456+0000 7fc39e7fc700 1 --2- 172.21.15.123:0/1387195409 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc38802ff60 0x7fc388032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:18.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.456+0000 7fc39e7fc700 1 --2- 172.21.15.123:0/1387195409 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc38802ff60 0x7fc388032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fc3900096a0 tx=0x7fc390006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:18.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.460+0000 7fc3877fe700 1 -- 172.21.15.123:0/1387195409 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc3940095f0 con 0x7fc3a00f5410 2024-01-13T11:58:18.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:18 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:17.583162+0000 mgr.smithi059.hmyldj (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:18.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.666+0000 7fc3a5884700 1 -- 172.21.15.123:0/1387195409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc3a0045ad0 con 0x7fc3a00f5410 2024-01-13T11:58:18.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.666+0000 7fc3877fe700 1 -- 172.21.15.123:0/1387195409 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc39401db70 con 0x7fc3a00f5410 2024-01-13T11:58:18.654 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:58:18.654 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:58:18.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.668+0000 7fc3a5884700 1 -- 172.21.15.123:0/1387195409 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc38802ff60 msgr2=0x7fc388032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:18.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.668+0000 7fc3a5884700 1 --2- 172.21.15.123:0/1387195409 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc38802ff60 0x7fc388032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fc3900096a0 tx=0x7fc390006b40).stop 2024-01-13T11:58:18.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.668+0000 7fc3a5884700 1 -- 172.21.15.123:0/1387195409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3a00f5410 msgr2=0x7fc3a0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:18.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.668+0000 7fc3a5884700 1 --2- 172.21.15.123:0/1387195409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3a00f5410 0x7fc3a0109450 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fc394013510 tx=0x7fc39400b040).stop 2024-01-13T11:58:18.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.668+0000 7fc3a5884700 1 -- 172.21.15.123:0/1387195409 shutdown_connections 2024-01-13T11:58:18.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.668+0000 7fc3a5884700 1 --2- 172.21.15.123:0/1387195409 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fc38802ff60 0x7fc388032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:18.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.668+0000 7fc3a5884700 1 --2- 172.21.15.123:0/1387195409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3a00f5410 0x7fc3a0109450 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:18.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.668+0000 7fc3a5884700 1 -- 172.21.15.123:0/1387195409 >> 172.21.15.123:0/1387195409 conn(0x7fc3a00f0b70 msgr2=0x7fc3a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:18.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.668+0000 7fc3a5884700 1 -- 172.21.15.123:0/1387195409 shutdown_connections 2024-01-13T11:58:18.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:18.668+0000 7fc3a5884700 1 -- 172.21.15.123:0/1387195409 wait complete. 2024-01-13T11:58:18.656 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:58:19.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:19 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:58:18.667724+0000 mon.smithi059 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.123:0/1387195409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:58:20.049 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:58:20.050 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:58:20.635 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:20 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:19.583542+0000 mgr.smithi059.hmyldj (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:21.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.212+0000 7f8adae90700 1 -- 172.21.15.123:0/3939893854 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ad40f5410 msgr2=0x7f8ad40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:21.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.212+0000 7f8adae90700 1 --2- 172.21.15.123:0/3939893854 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ad40f5410 0x7f8ad40f5830 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f8ac8009fb0 tx=0x7f8ac8009b30).stop 2024-01-13T11:58:21.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.212+0000 7f8adae90700 1 -- 172.21.15.123:0/3939893854 shutdown_connections 2024-01-13T11:58:21.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.212+0000 7f8adae90700 1 --2- 172.21.15.123:0/3939893854 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ad40f5410 0x7f8ad40f5830 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:21.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.212+0000 7f8adae90700 1 -- 172.21.15.123:0/3939893854 >> 172.21.15.123:0/3939893854 conn(0x7f8ad40f0b70 msgr2=0x7f8ad40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:21.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.212+0000 7f8adae90700 1 -- 172.21.15.123:0/3939893854 shutdown_connections 2024-01-13T11:58:21.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.212+0000 7f8adae90700 1 -- 172.21.15.123:0/3939893854 wait complete. 2024-01-13T11:58:21.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.213+0000 7f8adae90700 1 Processor -- start 2024-01-13T11:58:21.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.213+0000 7f8adae90700 1 -- start start 2024-01-13T11:58:21.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.213+0000 7f8adae90700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ad40f5410 0x7f8ad41055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:21.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.213+0000 7f8adae90700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad4105ac0 con 0x7f8ad40f5410 2024-01-13T11:58:21.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.214+0000 7f8ad8c2c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ad40f5410 0x7f8ad41055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:21.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.214+0000 7f8ad8c2c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ad40f5410 0x7f8ad41055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:37702/0 (socket says 172.21.15.123:37702) 2024-01-13T11:58:21.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.214+0000 7f8ad8c2c700 1 -- 172.21.15.123:0/3357362451 learned_addr learned my addr 172.21.15.123:0/3357362451 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:58:21.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.214+0000 7f8ad8c2c700 1 -- 172.21.15.123:0/3357362451 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac8005040 con 0x7f8ad40f5410 2024-01-13T11:58:21.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.214+0000 7f8ad8c2c700 1 --2- 172.21.15.123:0/3357362451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ad40f5410 0x7f8ad41055d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f8ac8013510 tx=0x7f8ac800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:21.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.214+0000 7f8ad17fa700 1 -- 172.21.15.123:0/3357362451 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ac8009320 con 0x7f8ad40f5410 2024-01-13T11:58:21.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.214+0000 7f8adae90700 1 -- 172.21.15.123:0/3357362451 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad4105cc0 con 0x7f8ad40f5410 2024-01-13T11:58:21.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.214+0000 7f8ad17fa700 1 -- 172.21.15.123:0/3357362451 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ac8013f00 con 0x7f8ad40f5410 2024-01-13T11:58:21.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.214+0000 7f8ad17fa700 1 -- 172.21.15.123:0/3357362451 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ac8014d90 con 0x7f8ad40f5410 2024-01-13T11:58:21.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.214+0000 7f8adae90700 1 -- 172.21.15.123:0/3357362451 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad4106150 con 0x7f8ad40f5410 2024-01-13T11:58:21.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.215+0000 7f8ad17fa700 1 -- 172.21.15.123:0/3357362451 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ac8014420 con 0x7f8ad40f5410 2024-01-13T11:58:21.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.215+0000 7f8adae90700 1 -- 172.21.15.123:0/3357362451 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad4045ad0 con 0x7f8ad40f5410 2024-01-13T11:58:21.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.215+0000 7f8ad17fa700 1 --2- 172.21.15.123:0/3357362451 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8ac402ff60 0x7f8ac4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:21.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.215+0000 7f8ad17fa700 1 -- 172.21.15.123:0/3357362451 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ac8041000 con 0x7f8ad40f5410 2024-01-13T11:58:21.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.215+0000 7f8ad3fff700 1 --2- 172.21.15.123:0/3357362451 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8ac402ff60 0x7f8ac4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:21.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.216+0000 7f8ad3fff700 1 --2- 172.21.15.123:0/3357362451 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8ac402ff60 0x7f8ac4032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f8ac00096a0 tx=0x7f8ac0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:21.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.220+0000 7f8ad17fa700 1 -- 172.21.15.123:0/3357362451 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ac80095f0 con 0x7f8ad40f5410 2024-01-13T11:58:21.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.429+0000 7f8adae90700 1 -- 172.21.15.123:0/3357362451 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ad4107000 con 0x7f8ad40f5410 2024-01-13T11:58:21.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.430+0000 7f8ad17fa700 1 -- 172.21.15.123:0/3357362451 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8ac801db70 con 0x7f8ad40f5410 2024-01-13T11:58:21.417 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:58:21.417 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:58:21.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.431+0000 7f8adae90700 1 -- 172.21.15.123:0/3357362451 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8ac402ff60 msgr2=0x7f8ac4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:21.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.431+0000 7f8adae90700 1 --2- 172.21.15.123:0/3357362451 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8ac402ff60 0x7f8ac4032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f8ac00096a0 tx=0x7f8ac0006b40).stop 2024-01-13T11:58:21.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.432+0000 7f8adae90700 1 -- 172.21.15.123:0/3357362451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ad40f5410 msgr2=0x7f8ad41055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:21.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.432+0000 7f8adae90700 1 --2- 172.21.15.123:0/3357362451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ad40f5410 0x7f8ad41055d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f8ac8013510 tx=0x7f8ac800b040).stop 2024-01-13T11:58:21.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.432+0000 7f8adae90700 1 -- 172.21.15.123:0/3357362451 shutdown_connections 2024-01-13T11:58:21.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.432+0000 7f8adae90700 1 --2- 172.21.15.123:0/3357362451 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f8ac402ff60 0x7f8ac4032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:21.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.432+0000 7f8adae90700 1 --2- 172.21.15.123:0/3357362451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ad40f5410 0x7f8ad41055d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:21.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.432+0000 7f8adae90700 1 -- 172.21.15.123:0/3357362451 >> 172.21.15.123:0/3357362451 conn(0x7f8ad40f0b70 msgr2=0x7f8ad40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:21.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.432+0000 7f8adae90700 1 -- 172.21.15.123:0/3357362451 shutdown_connections 2024-01-13T11:58:21.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:21.432+0000 7f8adae90700 1 -- 172.21.15.123:0/3357362451 wait complete. 2024-01-13T11:58:21.420 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:58:22.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:22 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:58:21.431397+0000 mon.smithi059 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.123:0/3357362451' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:58:22.805 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:58:22.805 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:58:23.635 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:23 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:21.583931+0000 mgr.smithi059.hmyldj (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:23.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.982+0000 7fed4550f700 1 -- 172.21.15.123:0/3170912042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed400f5410 msgr2=0x7fed400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:23.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.982+0000 7fed4550f700 1 --2- 172.21.15.123:0/3170912042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed400f5410 0x7fed400f5830 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fed34009fb0 tx=0x7fed34009b30).stop 2024-01-13T11:58:23.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.982+0000 7fed4550f700 1 -- 172.21.15.123:0/3170912042 shutdown_connections 2024-01-13T11:58:23.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.982+0000 7fed4550f700 1 --2- 172.21.15.123:0/3170912042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed400f5410 0x7fed400f5830 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:23.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.982+0000 7fed4550f700 1 -- 172.21.15.123:0/3170912042 >> 172.21.15.123:0/3170912042 conn(0x7fed400f0b70 msgr2=0x7fed400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:23.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.982+0000 7fed4550f700 1 -- 172.21.15.123:0/3170912042 shutdown_connections 2024-01-13T11:58:23.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.982+0000 7fed4550f700 1 -- 172.21.15.123:0/3170912042 wait complete. 2024-01-13T11:58:23.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.983+0000 7fed4550f700 1 Processor -- start 2024-01-13T11:58:23.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.983+0000 7fed4550f700 1 -- start start 2024-01-13T11:58:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.983+0000 7fed4550f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed400f5410 0x7fed4010b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.983+0000 7fed4550f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed401071e0 con 0x7fed400f5410 2024-01-13T11:58:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.983+0000 7fed3effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed400f5410 0x7fed4010b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.983+0000 7fed3effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed400f5410 0x7fed4010b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:37724/0 (socket says 172.21.15.123:37724) 2024-01-13T11:58:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.983+0000 7fed3effd700 1 -- 172.21.15.123:0/1891299097 learned_addr learned my addr 172.21.15.123:0/1891299097 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:58:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.984+0000 7fed3effd700 1 -- 172.21.15.123:0/1891299097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed34005040 con 0x7fed400f5410 2024-01-13T11:58:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.984+0000 7fed3effd700 1 --2- 172.21.15.123:0/1891299097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed400f5410 0x7fed4010b7f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fed34013510 tx=0x7fed3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.984+0000 7fed277fe700 1 -- 172.21.15.123:0/1891299097 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed34009320 con 0x7fed400f5410 2024-01-13T11:58:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.984+0000 7fed4550f700 1 -- 172.21.15.123:0/1891299097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed40060220 con 0x7fed400f5410 2024-01-13T11:58:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.984+0000 7fed277fe700 1 -- 172.21.15.123:0/1891299097 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed34013f00 con 0x7fed400f5410 2024-01-13T11:58:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.984+0000 7fed277fe700 1 -- 172.21.15.123:0/1891299097 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed34014d90 con 0x7fed400f5410 2024-01-13T11:58:23.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.984+0000 7fed4550f700 1 -- 172.21.15.123:0/1891299097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed4010bce0 con 0x7fed400f5410 2024-01-13T11:58:23.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.985+0000 7fed277fe700 1 -- 172.21.15.123:0/1891299097 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed34014420 con 0x7fed400f5410 2024-01-13T11:58:23.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.985+0000 7fed4550f700 1 -- 172.21.15.123:0/1891299097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed40045ad0 con 0x7fed400f5410 2024-01-13T11:58:23.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.985+0000 7fed277fe700 1 --2- 172.21.15.123:0/1891299097 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fed2802ff60 0x7fed28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:23.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.985+0000 7fed277fe700 1 -- 172.21.15.123:0/1891299097 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed34040fb0 con 0x7fed400f5410 2024-01-13T11:58:23.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.985+0000 7fed3e7fc700 1 --2- 172.21.15.123:0/1891299097 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fed2802ff60 0x7fed28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:23.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.986+0000 7fed3e7fc700 1 --2- 172.21.15.123:0/1891299097 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fed2802ff60 0x7fed28032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fed300096a0 tx=0x7fed30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:23.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:23.990+0000 7fed277fe700 1 -- 172.21.15.123:0/1891299097 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed340095f0 con 0x7fed400f5410 2024-01-13T11:58:24.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:24.198+0000 7fed4550f700 1 -- 172.21.15.123:0/1891299097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed4010c4a0 con 0x7fed400f5410 2024-01-13T11:58:24.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:24.199+0000 7fed277fe700 1 -- 172.21.15.123:0/1891299097 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fed3401db70 con 0x7fed400f5410 2024-01-13T11:58:24.186 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:58:24.186 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:58:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:24.200+0000 7fed4550f700 1 -- 172.21.15.123:0/1891299097 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fed2802ff60 msgr2=0x7fed28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:24.200+0000 7fed4550f700 1 --2- 172.21.15.123:0/1891299097 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fed2802ff60 0x7fed28032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fed300096a0 tx=0x7fed30006b40).stop 2024-01-13T11:58:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:24.200+0000 7fed4550f700 1 -- 172.21.15.123:0/1891299097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed400f5410 msgr2=0x7fed4010b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:24.200+0000 7fed4550f700 1 --2- 172.21.15.123:0/1891299097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed400f5410 0x7fed4010b7f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fed34013510 tx=0x7fed3400b040).stop 2024-01-13T11:58:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:24.200+0000 7fed4550f700 1 -- 172.21.15.123:0/1891299097 shutdown_connections 2024-01-13T11:58:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:24.200+0000 7fed4550f700 1 --2- 172.21.15.123:0/1891299097 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fed2802ff60 0x7fed28032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:24.200+0000 7fed4550f700 1 --2- 172.21.15.123:0/1891299097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed400f5410 0x7fed4010b7f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:24.200+0000 7fed4550f700 1 -- 172.21.15.123:0/1891299097 >> 172.21.15.123:0/1891299097 conn(0x7fed400f0b70 msgr2=0x7fed400f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:24.200+0000 7fed4550f700 1 -- 172.21.15.123:0/1891299097 shutdown_connections 2024-01-13T11:58:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:24.200+0000 7fed4550f700 1 -- 172.21.15.123:0/1891299097 wait complete. 2024-01-13T11:58:24.189 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:58:24.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:24 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:58:24.200216+0000 mon.smithi059 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.123:0/1891299097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:58:25.562 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:58:25.563 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:58:25.635 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:25 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:23.584287+0000 mgr.smithi059.hmyldj (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:26.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.744+0000 7ff3d9868700 1 -- 172.21.15.123:0/2775374183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3d40f47c0 msgr2=0x7ff3d40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:26.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.744+0000 7ff3d9868700 1 --2- 172.21.15.123:0/2775374183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3d40f47c0 0x7ff3d40f4be0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7ff3c4009fb0 tx=0x7ff3c4009b30).stop 2024-01-13T11:58:26.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.744+0000 7ff3d9868700 1 -- 172.21.15.123:0/2775374183 shutdown_connections 2024-01-13T11:58:26.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.744+0000 7ff3d9868700 1 --2- 172.21.15.123:0/2775374183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3d40f47c0 0x7ff3d40f4be0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:26.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.744+0000 7ff3d9868700 1 -- 172.21.15.123:0/2775374183 >> 172.21.15.123:0/2775374183 conn(0x7ff3d40f02e0 msgr2=0x7ff3d40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:26.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.744+0000 7ff3d9868700 1 -- 172.21.15.123:0/2775374183 shutdown_connections 2024-01-13T11:58:26.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.744+0000 7ff3d9868700 1 -- 172.21.15.123:0/2775374183 wait complete. 2024-01-13T11:58:26.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.744+0000 7ff3d9868700 1 Processor -- start 2024-01-13T11:58:26.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.745+0000 7ff3d9868700 1 -- start start 2024-01-13T11:58:26.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.745+0000 7ff3d9868700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3d40f47c0 0x7ff3d4106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:26.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.745+0000 7ff3d9868700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d4106bf0 con 0x7ff3d40f47c0 2024-01-13T11:58:26.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.745+0000 7ff3d2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3d40f47c0 0x7ff3d4106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:26.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.745+0000 7ff3d2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3d40f47c0 0x7ff3d4106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:40848/0 (socket says 172.21.15.123:40848) 2024-01-13T11:58:26.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.745+0000 7ff3d2ffd700 1 -- 172.21.15.123:0/1724912265 learned_addr learned my addr 172.21.15.123:0/1724912265 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:58:26.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.745+0000 7ff3d2ffd700 1 -- 172.21.15.123:0/1724912265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c4005040 con 0x7ff3d40f47c0 2024-01-13T11:58:26.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.745+0000 7ff3d2ffd700 1 --2- 172.21.15.123:0/1724912265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3d40f47c0 0x7ff3d4106700 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7ff3c4014000 tx=0x7ff3c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:26.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.745+0000 7ff3bb7fe700 1 -- 172.21.15.123:0/1724912265 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3c40095f0 con 0x7ff3d40f47c0 2024-01-13T11:58:26.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.745+0000 7ff3bb7fe700 1 -- 172.21.15.123:0/1724912265 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3c4007e70 con 0x7ff3d40f47c0 2024-01-13T11:58:26.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.745+0000 7ff3d9868700 1 -- 172.21.15.123:0/1724912265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d4106df0 con 0x7ff3d40f47c0 2024-01-13T11:58:26.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.745+0000 7ff3bb7fe700 1 -- 172.21.15.123:0/1724912265 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3c4014ea0 con 0x7ff3d40f47c0 2024-01-13T11:58:26.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.746+0000 7ff3d9868700 1 -- 172.21.15.123:0/1724912265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d4107280 con 0x7ff3d40f47c0 2024-01-13T11:58:26.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.746+0000 7ff3bb7fe700 1 -- 172.21.15.123:0/1724912265 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3c4025070 con 0x7ff3d40f47c0 2024-01-13T11:58:26.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.746+0000 7ff3d9868700 1 -- 172.21.15.123:0/1724912265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d4045ad0 con 0x7ff3d40f47c0 2024-01-13T11:58:26.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.746+0000 7ff3bb7fe700 1 --2- 172.21.15.123:0/1724912265 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff3bc02ffb0 0x7ff3bc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:26.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.747+0000 7ff3bb7fe700 1 -- 172.21.15.123:0/1724912265 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff3c4040a30 con 0x7ff3d40f47c0 2024-01-13T11:58:26.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.747+0000 7ff3d27fc700 1 --2- 172.21.15.123:0/1724912265 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff3bc02ffb0 0x7ff3bc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:26.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.747+0000 7ff3d27fc700 1 --2- 172.21.15.123:0/1724912265 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff3bc02ffb0 0x7ff3bc032470 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7ff3c80096a0 tx=0x7ff3c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:26.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.751+0000 7ff3bb7fe700 1 -- 172.21.15.123:0/1724912265 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3c40079c0 con 0x7ff3d40f47c0 2024-01-13T11:58:26.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.957+0000 7ff3d9868700 1 -- 172.21.15.123:0/1724912265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3d4108130 con 0x7ff3d40f47c0 2024-01-13T11:58:26.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.958+0000 7ff3bb7fe700 1 -- 172.21.15.123:0/1724912265 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff3c400f300 con 0x7ff3d40f47c0 2024-01-13T11:58:26.945 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:58:26.945 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:58:26.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.959+0000 7ff3d9868700 1 -- 172.21.15.123:0/1724912265 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff3bc02ffb0 msgr2=0x7ff3bc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:26.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.959+0000 7ff3d9868700 1 --2- 172.21.15.123:0/1724912265 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff3bc02ffb0 0x7ff3bc032470 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7ff3c80096a0 tx=0x7ff3c8006b40).stop 2024-01-13T11:58:26.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.959+0000 7ff3d9868700 1 -- 172.21.15.123:0/1724912265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3d40f47c0 msgr2=0x7ff3d4106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:26.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.959+0000 7ff3d9868700 1 --2- 172.21.15.123:0/1724912265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3d40f47c0 0x7ff3d4106700 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7ff3c4014000 tx=0x7ff3c400b040).stop 2024-01-13T11:58:26.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.959+0000 7ff3d9868700 1 -- 172.21.15.123:0/1724912265 shutdown_connections 2024-01-13T11:58:26.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.959+0000 7ff3d9868700 1 --2- 172.21.15.123:0/1724912265 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7ff3bc02ffb0 0x7ff3bc032470 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:26.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.959+0000 7ff3d9868700 1 --2- 172.21.15.123:0/1724912265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3d40f47c0 0x7ff3d4106700 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:26.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.959+0000 7ff3d9868700 1 -- 172.21.15.123:0/1724912265 >> 172.21.15.123:0/1724912265 conn(0x7ff3d40f02e0 msgr2=0x7ff3d40f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:26.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.959+0000 7ff3d9868700 1 -- 172.21.15.123:0/1724912265 shutdown_connections 2024-01-13T11:58:26.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:26.959+0000 7ff3d9868700 1 -- 172.21.15.123:0/1724912265 wait complete. 2024-01-13T11:58:26.948 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:58:27.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:27 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:25.584636+0000 mgr.smithi059.hmyldj (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:27.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:27 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:58:26.959256+0000 mon.smithi059 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.123:0/1724912265' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:58:28.347 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:58:28.347 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:58:29.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.501+0000 7f6a7872c700 1 -- 172.21.15.123:0/1114519401 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a700f5410 msgr2=0x7f6a700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:29.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.501+0000 7f6a7872c700 1 --2- 172.21.15.123:0/1114519401 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a700f5410 0x7f6a700f5830 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f6a60009fb0 tx=0x7f6a60009b30).stop 2024-01-13T11:58:29.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.501+0000 7f6a7872c700 1 -- 172.21.15.123:0/1114519401 shutdown_connections 2024-01-13T11:58:29.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.501+0000 7f6a7872c700 1 --2- 172.21.15.123:0/1114519401 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a700f5410 0x7f6a700f5830 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:29.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.501+0000 7f6a7872c700 1 -- 172.21.15.123:0/1114519401 >> 172.21.15.123:0/1114519401 conn(0x7f6a700f0b70 msgr2=0x7f6a700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:29.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.501+0000 7f6a7872c700 1 -- 172.21.15.123:0/1114519401 shutdown_connections 2024-01-13T11:58:29.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.501+0000 7f6a7872c700 1 -- 172.21.15.123:0/1114519401 wait complete. 2024-01-13T11:58:29.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.502+0000 7f6a7872c700 1 Processor -- start 2024-01-13T11:58:29.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.502+0000 7f6a7872c700 1 -- start start 2024-01-13T11:58:29.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.502+0000 7f6a7872c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a700f5410 0x7f6a70109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:29.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.502+0000 7f6a7872c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a70060240 con 0x7f6a700f5410 2024-01-13T11:58:29.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.502+0000 7f6a764c8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a700f5410 0x7f6a70109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:29.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.502+0000 7f6a764c8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a700f5410 0x7f6a70109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:40860/0 (socket says 172.21.15.123:40860) 2024-01-13T11:58:29.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.502+0000 7f6a764c8700 1 -- 172.21.15.123:0/250036267 learned_addr learned my addr 172.21.15.123:0/250036267 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:58:29.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.503+0000 7f6a764c8700 1 -- 172.21.15.123:0/250036267 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a60005040 con 0x7f6a700f5410 2024-01-13T11:58:29.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.503+0000 7f6a764c8700 1 --2- 172.21.15.123:0/250036267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a700f5410 0x7f6a70109430 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f6a60013510 tx=0x7f6a6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:29.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.503+0000 7f6a66ffd700 1 -- 172.21.15.123:0/250036267 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a60009320 con 0x7f6a700f5410 2024-01-13T11:58:29.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.503+0000 7f6a7872c700 1 -- 172.21.15.123:0/250036267 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a70060440 con 0x7f6a700f5410 2024-01-13T11:58:29.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.503+0000 7f6a66ffd700 1 -- 172.21.15.123:0/250036267 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a60013f00 con 0x7f6a700f5410 2024-01-13T11:58:29.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.503+0000 7f6a66ffd700 1 -- 172.21.15.123:0/250036267 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a60014d90 con 0x7f6a700f5410 2024-01-13T11:58:29.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.503+0000 7f6a7872c700 1 -- 172.21.15.123:0/250036267 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a70109ae0 con 0x7f6a700f5410 2024-01-13T11:58:29.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.504+0000 7f6a66ffd700 1 -- 172.21.15.123:0/250036267 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6a600095f0 con 0x7f6a700f5410 2024-01-13T11:58:29.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.504+0000 7f6a7872c700 1 -- 172.21.15.123:0/250036267 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a70045ad0 con 0x7f6a700f5410 2024-01-13T11:58:29.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.504+0000 7f6a66ffd700 1 --2- 172.21.15.123:0/250036267 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6a5c02ff60 0x7f6a5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:29.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.504+0000 7f6a66ffd700 1 -- 172.21.15.123:0/250036267 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6a60041000 con 0x7f6a700f5410 2024-01-13T11:58:29.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.504+0000 7f6a75cc7700 1 --2- 172.21.15.123:0/250036267 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6a5c02ff60 0x7f6a5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:29.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.505+0000 7f6a75cc7700 1 --2- 172.21.15.123:0/250036267 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6a5c02ff60 0x7f6a5c032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f6a6c0096a0 tx=0x7f6a6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:29.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.509+0000 7f6a66ffd700 1 -- 172.21.15.123:0/250036267 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a60014420 con 0x7f6a700f5410 2024-01-13T11:58:29.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:29 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:27.585023+0000 mgr.smithi059.hmyldj (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:29.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.717+0000 7f6a7872c700 1 -- 172.21.15.123:0/250036267 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a700fae60 con 0x7f6a700f5410 2024-01-13T11:58:29.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.718+0000 7f6a66ffd700 1 -- 172.21.15.123:0/250036267 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6a6001d7f0 con 0x7f6a700f5410 2024-01-13T11:58:29.706 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:58:29.706 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:58:29.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.719+0000 7f6a7872c700 1 -- 172.21.15.123:0/250036267 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6a5c02ff60 msgr2=0x7f6a5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:29.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.719+0000 7f6a7872c700 1 --2- 172.21.15.123:0/250036267 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6a5c02ff60 0x7f6a5c032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f6a6c0096a0 tx=0x7f6a6c006b40).stop 2024-01-13T11:58:29.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.719+0000 7f6a7872c700 1 -- 172.21.15.123:0/250036267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a700f5410 msgr2=0x7f6a70109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:29.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.720+0000 7f6a7872c700 1 --2- 172.21.15.123:0/250036267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a700f5410 0x7f6a70109430 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f6a60013510 tx=0x7f6a6000b040).stop 2024-01-13T11:58:29.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.720+0000 7f6a7872c700 1 -- 172.21.15.123:0/250036267 shutdown_connections 2024-01-13T11:58:29.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.720+0000 7f6a7872c700 1 --2- 172.21.15.123:0/250036267 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f6a5c02ff60 0x7f6a5c032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:29.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.720+0000 7f6a7872c700 1 --2- 172.21.15.123:0/250036267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a700f5410 0x7f6a70109430 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:29.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.720+0000 7f6a7872c700 1 -- 172.21.15.123:0/250036267 >> 172.21.15.123:0/250036267 conn(0x7f6a700f0b70 msgr2=0x7f6a700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:29.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.720+0000 7f6a7872c700 1 -- 172.21.15.123:0/250036267 shutdown_connections 2024-01-13T11:58:29.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:29.720+0000 7f6a7872c700 1 -- 172.21.15.123:0/250036267 wait complete. 2024-01-13T11:58:29.708 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:58:30.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:30 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:58:29.719484+0000 mon.smithi059 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.123:0/250036267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:58:31.091 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:58:31.092 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:58:31.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:31 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:29.585391+0000 mgr.smithi059.hmyldj (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:32.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.413+0000 7fe46874c700 1 -- 172.21.15.123:0/1018467017 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4600f5410 msgr2=0x7fe4600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:32.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.413+0000 7fe46874c700 1 --2- 172.21.15.123:0/1018467017 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4600f5410 0x7fe4600f5830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fe450009fb0 tx=0x7fe450009b30).stop 2024-01-13T11:58:32.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.413+0000 7fe46874c700 1 -- 172.21.15.123:0/1018467017 shutdown_connections 2024-01-13T11:58:32.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.413+0000 7fe46874c700 1 --2- 172.21.15.123:0/1018467017 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4600f5410 0x7fe4600f5830 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:32.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.413+0000 7fe46874c700 1 -- 172.21.15.123:0/1018467017 >> 172.21.15.123:0/1018467017 conn(0x7fe4600f0b70 msgr2=0x7fe4600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:32.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.413+0000 7fe46874c700 1 -- 172.21.15.123:0/1018467017 shutdown_connections 2024-01-13T11:58:32.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.413+0000 7fe46874c700 1 -- 172.21.15.123:0/1018467017 wait complete. 2024-01-13T11:58:32.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.414+0000 7fe46874c700 1 Processor -- start 2024-01-13T11:58:32.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.414+0000 7fe46874c700 1 -- start start 2024-01-13T11:58:32.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.414+0000 7fe46874c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4600f5410 0x7fe460109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:32.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.414+0000 7fe46874c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe460060240 con 0x7fe4600f5410 2024-01-13T11:58:32.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.415+0000 7fe4664e8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4600f5410 0x7fe460109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:32.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.415+0000 7fe4664e8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4600f5410 0x7fe460109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:40874/0 (socket says 172.21.15.123:40874) 2024-01-13T11:58:32.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.415+0000 7fe4664e8700 1 -- 172.21.15.123:0/1508457370 learned_addr learned my addr 172.21.15.123:0/1508457370 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:58:32.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.415+0000 7fe4664e8700 1 -- 172.21.15.123:0/1508457370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe450005040 con 0x7fe4600f5410 2024-01-13T11:58:32.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.415+0000 7fe4664e8700 1 --2- 172.21.15.123:0/1508457370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4600f5410 0x7fe460109430 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fe450013510 tx=0x7fe45000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:32.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.415+0000 7fe456ffd700 1 -- 172.21.15.123:0/1508457370 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe450009320 con 0x7fe4600f5410 2024-01-13T11:58:32.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.415+0000 7fe46874c700 1 -- 172.21.15.123:0/1508457370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe460060440 con 0x7fe4600f5410 2024-01-13T11:58:32.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.415+0000 7fe456ffd700 1 -- 172.21.15.123:0/1508457370 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe450013f00 con 0x7fe4600f5410 2024-01-13T11:58:32.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.415+0000 7fe456ffd700 1 -- 172.21.15.123:0/1508457370 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe450014d90 con 0x7fe4600f5410 2024-01-13T11:58:32.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.415+0000 7fe46874c700 1 -- 172.21.15.123:0/1508457370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe460109ae0 con 0x7fe4600f5410 2024-01-13T11:58:32.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.416+0000 7fe456ffd700 1 -- 172.21.15.123:0/1508457370 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe4500095f0 con 0x7fe4600f5410 2024-01-13T11:58:32.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.416+0000 7fe456ffd700 1 --2- 172.21.15.123:0/1508457370 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe44c02ff60 0x7fe44c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:32.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.416+0000 7fe46874c700 1 -- 172.21.15.123:0/1508457370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe460045ad0 con 0x7fe4600f5410 2024-01-13T11:58:32.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.416+0000 7fe456ffd700 1 -- 172.21.15.123:0/1508457370 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe450041000 con 0x7fe4600f5410 2024-01-13T11:58:32.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.417+0000 7fe465ce7700 1 --2- 172.21.15.123:0/1508457370 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe44c02ff60 0x7fe44c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:32.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.417+0000 7fe465ce7700 1 --2- 172.21.15.123:0/1508457370 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe44c02ff60 0x7fe44c032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fe45c0096a0 tx=0x7fe45c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:32.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.421+0000 7fe456ffd700 1 -- 172.21.15.123:0/1508457370 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe450014420 con 0x7fe4600f5410 2024-01-13T11:58:32.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.625+0000 7fe46874c700 1 -- 172.21.15.123:0/1508457370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4600faf50 con 0x7fe4600f5410 2024-01-13T11:58:32.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.626+0000 7fe456ffd700 1 -- 172.21.15.123:0/1508457370 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe45002b020 con 0x7fe4600f5410 2024-01-13T11:58:32.613 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:58:32.614 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:58:32.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.628+0000 7fe46874c700 1 -- 172.21.15.123:0/1508457370 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe44c02ff60 msgr2=0x7fe44c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:32.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.628+0000 7fe46874c700 1 --2- 172.21.15.123:0/1508457370 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe44c02ff60 0x7fe44c032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fe45c0096a0 tx=0x7fe45c006b40).stop 2024-01-13T11:58:32.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.628+0000 7fe46874c700 1 -- 172.21.15.123:0/1508457370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4600f5410 msgr2=0x7fe460109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:32.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.628+0000 7fe46874c700 1 --2- 172.21.15.123:0/1508457370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4600f5410 0x7fe460109430 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fe450013510 tx=0x7fe45000b040).stop 2024-01-13T11:58:32.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.628+0000 7fe46874c700 1 -- 172.21.15.123:0/1508457370 shutdown_connections 2024-01-13T11:58:32.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.628+0000 7fe46874c700 1 --2- 172.21.15.123:0/1508457370 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fe44c02ff60 0x7fe44c032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:32.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.628+0000 7fe46874c700 1 --2- 172.21.15.123:0/1508457370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe4600f5410 0x7fe460109430 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:32.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.628+0000 7fe46874c700 1 -- 172.21.15.123:0/1508457370 >> 172.21.15.123:0/1508457370 conn(0x7fe4600f0b70 msgr2=0x7fe4600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:32.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.628+0000 7fe46874c700 1 -- 172.21.15.123:0/1508457370 shutdown_connections 2024-01-13T11:58:32.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:32.628+0000 7fe46874c700 1 -- 172.21.15.123:0/1508457370 wait complete. 2024-01-13T11:58:32.616 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:58:33.635 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:33 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:31.585811+0000 mgr.smithi059.hmyldj (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:33.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:33 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:58:32.627482+0000 mon.smithi059 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.123:0/1508457370' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:58:34.012 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:58:34.012 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:58:35.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.184+0000 7f0cd58c3700 1 -- 172.21.15.123:0/125188269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cd00f62a0 msgr2=0x7f0cd00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:35.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.184+0000 7f0cd58c3700 1 --2- 172.21.15.123:0/125188269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cd00f62a0 0x7f0cd00f66c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f0cc4009fb0 tx=0x7f0cc4009b30).stop 2024-01-13T11:58:35.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.184+0000 7f0cd58c3700 1 -- 172.21.15.123:0/125188269 shutdown_connections 2024-01-13T11:58:35.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.184+0000 7f0cd58c3700 1 --2- 172.21.15.123:0/125188269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cd00f62a0 0x7f0cd00f66c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:35.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.184+0000 7f0cd58c3700 1 -- 172.21.15.123:0/125188269 >> 172.21.15.123:0/125188269 conn(0x7f0cd00f1aa0 msgr2=0x7f0cd00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:35.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.184+0000 7f0cd58c3700 1 -- 172.21.15.123:0/125188269 shutdown_connections 2024-01-13T11:58:35.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.184+0000 7f0cd58c3700 1 -- 172.21.15.123:0/125188269 wait complete. 2024-01-13T11:58:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.185+0000 7f0cd58c3700 1 Processor -- start 2024-01-13T11:58:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.185+0000 7f0cd58c3700 1 -- start start 2024-01-13T11:58:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.185+0000 7f0cd58c3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cd00f62a0 0x7f0cd006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.185+0000 7f0cd58c3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd0069e20 con 0x7f0cd00f62a0 2024-01-13T11:58:35.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.185+0000 7f0cd48c1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cd00f62a0 0x7f0cd006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.185+0000 7f0cd48c1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cd00f62a0 0x7f0cd006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:40892/0 (socket says 172.21.15.123:40892) 2024-01-13T11:58:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.185+0000 7f0cd48c1700 1 -- 172.21.15.123:0/3911797905 learned_addr learned my addr 172.21.15.123:0/3911797905 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:58:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.186+0000 7f0cd48c1700 1 -- 172.21.15.123:0/3911797905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc4005040 con 0x7f0cd00f62a0 2024-01-13T11:58:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.186+0000 7f0cd48c1700 1 --2- 172.21.15.123:0/3911797905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cd00f62a0 0x7f0cd006b9a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f0cc4013510 tx=0x7f0cc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.186+0000 7f0ccd7fa700 1 -- 172.21.15.123:0/3911797905 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cc4009320 con 0x7f0cd00f62a0 2024-01-13T11:58:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.186+0000 7f0cd58c3700 1 -- 172.21.15.123:0/3911797905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd0069fc0 con 0x7f0cd00f62a0 2024-01-13T11:58:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.186+0000 7f0ccd7fa700 1 -- 172.21.15.123:0/3911797905 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cc4013f00 con 0x7f0cd00f62a0 2024-01-13T11:58:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.186+0000 7f0ccd7fa700 1 -- 172.21.15.123:0/3911797905 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cc4014dc0 con 0x7f0cd00f62a0 2024-01-13T11:58:35.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.186+0000 7f0cd58c3700 1 -- 172.21.15.123:0/3911797905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd006a450 con 0x7f0cd00f62a0 2024-01-13T11:58:35.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.187+0000 7f0ccd7fa700 1 -- 172.21.15.123:0/3911797905 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0cc401d3d0 con 0x7f0cd00f62a0 2024-01-13T11:58:35.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.187+0000 7f0cd58c3700 1 -- 172.21.15.123:0/3911797905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd0059100 con 0x7f0cd00f62a0 2024-01-13T11:58:35.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.187+0000 7f0ccd7fa700 1 --2- 172.21.15.123:0/3911797905 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0cc002ff60 0x7f0cc0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:35.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.187+0000 7f0ccd7fa700 1 -- 172.21.15.123:0/3911797905 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0cc4041320 con 0x7f0cd00f62a0 2024-01-13T11:58:35.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.187+0000 7f0ccffff700 1 --2- 172.21.15.123:0/3911797905 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0cc002ff60 0x7f0cc0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:35.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.187+0000 7f0ccffff700 1 --2- 172.21.15.123:0/3911797905 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0cc002ff60 0x7f0cc0032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f0cbc0096a0 tx=0x7f0cbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:35.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.191+0000 7f0ccd7fa700 1 -- 172.21.15.123:0/3911797905 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0cc40126f0 con 0x7f0cd00f62a0 2024-01-13T11:58:35.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.399+0000 7f0cd58c3700 1 -- 172.21.15.123:0/3911797905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0cd00fbce0 con 0x7f0cd00f62a0 2024-01-13T11:58:35.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.400+0000 7f0ccd7fa700 1 -- 172.21.15.123:0/3911797905 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0cc4012330 con 0x7f0cd00f62a0 2024-01-13T11:58:35.388 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:58:35.388 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:58:35.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.401+0000 7f0cd58c3700 1 -- 172.21.15.123:0/3911797905 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0cc002ff60 msgr2=0x7f0cc0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:35.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.401+0000 7f0cd58c3700 1 --2- 172.21.15.123:0/3911797905 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0cc002ff60 0x7f0cc0032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f0cbc0096a0 tx=0x7f0cbc006b40).stop 2024-01-13T11:58:35.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.401+0000 7f0cd58c3700 1 -- 172.21.15.123:0/3911797905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cd00f62a0 msgr2=0x7f0cd006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:35.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.401+0000 7f0cd58c3700 1 --2- 172.21.15.123:0/3911797905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cd00f62a0 0x7f0cd006b9a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f0cc4013510 tx=0x7f0cc400b040).stop 2024-01-13T11:58:35.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.401+0000 7f0cd58c3700 1 -- 172.21.15.123:0/3911797905 shutdown_connections 2024-01-13T11:58:35.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.401+0000 7f0cd58c3700 1 --2- 172.21.15.123:0/3911797905 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f0cc002ff60 0x7f0cc0032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:35.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.401+0000 7f0cd58c3700 1 --2- 172.21.15.123:0/3911797905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0cd00f62a0 0x7f0cd006b9a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:35.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.401+0000 7f0cd58c3700 1 -- 172.21.15.123:0/3911797905 >> 172.21.15.123:0/3911797905 conn(0x7f0cd00f1aa0 msgr2=0x7f0cd00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:35.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.401+0000 7f0cd58c3700 1 -- 172.21.15.123:0/3911797905 shutdown_connections 2024-01-13T11:58:35.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:35.402+0000 7f0cd58c3700 1 -- 172.21.15.123:0/3911797905 wait complete. 2024-01-13T11:58:35.390 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:58:35.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:35 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:33.586152+0000 mgr.smithi059.hmyldj (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:36.635 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:36 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:58:35.401085+0000 mon.smithi059 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.123:0/3911797905' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:58:36.803 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:58:36.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:58:37.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:37 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:35.586452+0000 mgr.smithi059.hmyldj (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:38.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.035+0000 7fb79ae62700 1 -- 172.21.15.123:0/1952860387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7940f5410 msgr2=0x7fb7940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:38.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.035+0000 7fb79ae62700 1 --2- 172.21.15.123:0/1952860387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7940f5410 0x7fb7940f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fb788009fb0 tx=0x7fb788009b30).stop 2024-01-13T11:58:38.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.035+0000 7fb79ae62700 1 -- 172.21.15.123:0/1952860387 shutdown_connections 2024-01-13T11:58:38.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.035+0000 7fb79ae62700 1 --2- 172.21.15.123:0/1952860387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7940f5410 0x7fb7940f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:38.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.035+0000 7fb79ae62700 1 -- 172.21.15.123:0/1952860387 >> 172.21.15.123:0/1952860387 conn(0x7fb7940f0b70 msgr2=0x7fb7940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:38.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.035+0000 7fb79ae62700 1 -- 172.21.15.123:0/1952860387 shutdown_connections 2024-01-13T11:58:38.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.035+0000 7fb79ae62700 1 -- 172.21.15.123:0/1952860387 wait complete. 2024-01-13T11:58:38.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.035+0000 7fb79ae62700 1 Processor -- start 2024-01-13T11:58:38.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.035+0000 7fb79ae62700 1 -- start start 2024-01-13T11:58:38.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.036+0000 7fb79ae62700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7940f5410 0x7fb794105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:38.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.036+0000 7fb79ae62700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb794105b20 con 0x7fb7940f5410 2024-01-13T11:58:38.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.036+0000 7fb798bfe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7940f5410 0x7fb794105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:38.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.036+0000 7fb798bfe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7940f5410 0x7fb794105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:54686/0 (socket says 172.21.15.123:54686) 2024-01-13T11:58:38.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.036+0000 7fb798bfe700 1 -- 172.21.15.123:0/3456146728 learned_addr learned my addr 172.21.15.123:0/3456146728 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:58:38.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.036+0000 7fb798bfe700 1 -- 172.21.15.123:0/3456146728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb788005040 con 0x7fb7940f5410 2024-01-13T11:58:38.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.036+0000 7fb798bfe700 1 --2- 172.21.15.123:0/3456146728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7940f5410 0x7fb794105630 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fb788013510 tx=0x7fb78800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:38.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.036+0000 7fb7917fa700 1 -- 172.21.15.123:0/3456146728 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb788009320 con 0x7fb7940f5410 2024-01-13T11:58:38.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.036+0000 7fb79ae62700 1 -- 172.21.15.123:0/3456146728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb794105d20 con 0x7fb7940f5410 2024-01-13T11:58:38.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.036+0000 7fb7917fa700 1 -- 172.21.15.123:0/3456146728 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb788013f00 con 0x7fb7940f5410 2024-01-13T11:58:38.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.036+0000 7fb7917fa700 1 -- 172.21.15.123:0/3456146728 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb788014d90 con 0x7fb7940f5410 2024-01-13T11:58:38.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.036+0000 7fb79ae62700 1 -- 172.21.15.123:0/3456146728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7941061b0 con 0x7fb7940f5410 2024-01-13T11:58:38.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.037+0000 7fb7917fa700 1 -- 172.21.15.123:0/3456146728 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb788014420 con 0x7fb7940f5410 2024-01-13T11:58:38.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.037+0000 7fb79ae62700 1 -- 172.21.15.123:0/3456146728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb794045ad0 con 0x7fb7940f5410 2024-01-13T11:58:38.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.037+0000 7fb7917fa700 1 --2- 172.21.15.123:0/3456146728 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb78402ff60 0x7fb784032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:38.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.037+0000 7fb7917fa700 1 -- 172.21.15.123:0/3456146728 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb788041110 con 0x7fb7940f5410 2024-01-13T11:58:38.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.038+0000 7fb793fff700 1 --2- 172.21.15.123:0/3456146728 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb78402ff60 0x7fb784032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:38.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.038+0000 7fb793fff700 1 --2- 172.21.15.123:0/3456146728 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb78402ff60 0x7fb784032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fb7800096a0 tx=0x7fb780006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:38.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.042+0000 7fb7917fa700 1 -- 172.21.15.123:0/3456146728 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb7880095f0 con 0x7fb7940f5410 2024-01-13T11:58:38.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.251+0000 7fb79ae62700 1 -- 172.21.15.123:0/3456146728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb794107000 con 0x7fb7940f5410 2024-01-13T11:58:38.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.252+0000 7fb7917fa700 1 -- 172.21.15.123:0/3456146728 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb78801db70 con 0x7fb7940f5410 2024-01-13T11:58:38.239 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:58:38.239 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:58:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.254+0000 7fb79ae62700 1 -- 172.21.15.123:0/3456146728 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb78402ff60 msgr2=0x7fb784032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.254+0000 7fb79ae62700 1 --2- 172.21.15.123:0/3456146728 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb78402ff60 0x7fb784032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fb7800096a0 tx=0x7fb780006b40).stop 2024-01-13T11:58:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.254+0000 7fb79ae62700 1 -- 172.21.15.123:0/3456146728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7940f5410 msgr2=0x7fb794105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.254+0000 7fb79ae62700 1 --2- 172.21.15.123:0/3456146728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7940f5410 0x7fb794105630 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fb788013510 tx=0x7fb78800b040).stop 2024-01-13T11:58:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.254+0000 7fb79ae62700 1 -- 172.21.15.123:0/3456146728 shutdown_connections 2024-01-13T11:58:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.254+0000 7fb79ae62700 1 --2- 172.21.15.123:0/3456146728 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7fb78402ff60 0x7fb784032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.254+0000 7fb79ae62700 1 --2- 172.21.15.123:0/3456146728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7940f5410 0x7fb794105630 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.254+0000 7fb79ae62700 1 -- 172.21.15.123:0/3456146728 >> 172.21.15.123:0/3456146728 conn(0x7fb7940f0b70 msgr2=0x7fb7940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.254+0000 7fb79ae62700 1 -- 172.21.15.123:0/3456146728 shutdown_connections 2024-01-13T11:58:38.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:38.254+0000 7fb79ae62700 1 -- 172.21.15.123:0/3456146728 wait complete. 2024-01-13T11:58:38.242 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:58:38.635 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:38 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:58:38.253551+0000 mon.smithi059 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.123:0/3456146728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:58:39.620 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:58:39.621 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:58:39.635 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:39 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:37.586785+0000 mgr.smithi059.hmyldj (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:40.635 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:58:39.646843+0000 mon.smithi059 (mon.0) 324 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/trash_purge_schedule"}]: dispatch 2024-01-13T11:58:40.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:40 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:58:39.647090+0000 mon.smithi059 (mon.0) 325 : audit [INF] from='mgr.14162 172.21.15.59:0/4196719448' entity='mgr.smithi059.hmyldj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.hmyldj/mirror_snapshot_schedule"}]: dispatch 2024-01-13T11:58:40.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.834+0000 7f9c59a49700 1 -- 172.21.15.123:0/68725701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c540f5410 msgr2=0x7f9c540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:40.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.834+0000 7f9c59a49700 1 --2- 172.21.15.123:0/68725701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c540f5410 0x7f9c540f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f9c44009fb0 tx=0x7f9c44009b30).stop 2024-01-13T11:58:40.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.834+0000 7f9c59a49700 1 -- 172.21.15.123:0/68725701 shutdown_connections 2024-01-13T11:58:40.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.834+0000 7f9c59a49700 1 --2- 172.21.15.123:0/68725701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c540f5410 0x7f9c540f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:40.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.834+0000 7f9c59a49700 1 -- 172.21.15.123:0/68725701 >> 172.21.15.123:0/68725701 conn(0x7f9c540f0b70 msgr2=0x7f9c540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:40.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.834+0000 7f9c59a49700 1 -- 172.21.15.123:0/68725701 shutdown_connections 2024-01-13T11:58:40.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.834+0000 7f9c59a49700 1 -- 172.21.15.123:0/68725701 wait complete. 2024-01-13T11:58:40.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.834+0000 7f9c59a49700 1 Processor -- start 2024-01-13T11:58:40.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.834+0000 7f9c59a49700 1 -- start start 2024-01-13T11:58:40.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.835+0000 7f9c59a49700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c540f5410 0x7f9c54106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:40.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.835+0000 7f9c59a49700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c54106bf0 con 0x7f9c540f5410 2024-01-13T11:58:40.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.835+0000 7f9c52ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c540f5410 0x7f9c54106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:40.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.835+0000 7f9c52ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c540f5410 0x7f9c54106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:54708/0 (socket says 172.21.15.123:54708) 2024-01-13T11:58:40.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.835+0000 7f9c52ffd700 1 -- 172.21.15.123:0/1961839249 learned_addr learned my addr 172.21.15.123:0/1961839249 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:58:40.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.835+0000 7f9c52ffd700 1 -- 172.21.15.123:0/1961839249 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c44005040 con 0x7f9c540f5410 2024-01-13T11:58:40.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.835+0000 7f9c52ffd700 1 --2- 172.21.15.123:0/1961839249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c540f5410 0x7f9c54106700 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f9c44013510 tx=0x7f9c4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:40.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.835+0000 7f9c3b7fe700 1 -- 172.21.15.123:0/1961839249 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c44009320 con 0x7f9c540f5410 2024-01-13T11:58:40.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.835+0000 7f9c3b7fe700 1 -- 172.21.15.123:0/1961839249 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c44013f00 con 0x7f9c540f5410 2024-01-13T11:58:40.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.835+0000 7f9c59a49700 1 -- 172.21.15.123:0/1961839249 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c54106df0 con 0x7f9c540f5410 2024-01-13T11:58:40.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.836+0000 7f9c3b7fe700 1 -- 172.21.15.123:0/1961839249 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c44014d90 con 0x7f9c540f5410 2024-01-13T11:58:40.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.836+0000 7f9c59a49700 1 -- 172.21.15.123:0/1961839249 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c54107280 con 0x7f9c540f5410 2024-01-13T11:58:40.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.836+0000 7f9c3b7fe700 1 -- 172.21.15.123:0/1961839249 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c440263a0 con 0x7f9c540f5410 2024-01-13T11:58:40.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.837+0000 7f9c3b7fe700 1 --2- 172.21.15.123:0/1961839249 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9c3c02ff60 0x7f9c3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:40.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.837+0000 7f9c3b7fe700 1 -- 172.21.15.123:0/1961839249 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c44041370 con 0x7f9c540f5410 2024-01-13T11:58:40.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.837+0000 7f9c59a49700 1 -- 172.21.15.123:0/1961839249 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c54059100 con 0x7f9c540f5410 2024-01-13T11:58:40.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.837+0000 7f9c527fc700 1 --2- 172.21.15.123:0/1961839249 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9c3c02ff60 0x7f9c3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:40.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.837+0000 7f9c527fc700 1 --2- 172.21.15.123:0/1961839249 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9c3c02ff60 0x7f9c3c032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f9c480096a0 tx=0x7f9c48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:40.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:40.841+0000 7f9c3b7fe700 1 -- 172.21.15.123:0/1961839249 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c44012330 con 0x7f9c540f5410 2024-01-13T11:58:41.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:41.047+0000 7f9c59a49700 1 -- 172.21.15.123:0/1961839249 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c54045ad0 con 0x7f9c540f5410 2024-01-13T11:58:41.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:41.048+0000 7f9c3b7fe700 1 -- 172.21.15.123:0/1961839249 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9c4400f340 con 0x7f9c540f5410 2024-01-13T11:58:41.035 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:58:41.035 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:58:41.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:41.049+0000 7f9c59a49700 1 -- 172.21.15.123:0/1961839249 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9c3c02ff60 msgr2=0x7f9c3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:41.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:41.049+0000 7f9c59a49700 1 --2- 172.21.15.123:0/1961839249 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9c3c02ff60 0x7f9c3c032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f9c480096a0 tx=0x7f9c48006b40).stop 2024-01-13T11:58:41.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:41.049+0000 7f9c59a49700 1 -- 172.21.15.123:0/1961839249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c540f5410 msgr2=0x7f9c54106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:41.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:41.049+0000 7f9c59a49700 1 --2- 172.21.15.123:0/1961839249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c540f5410 0x7f9c54106700 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f9c44013510 tx=0x7f9c4400b040).stop 2024-01-13T11:58:41.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:41.049+0000 7f9c59a49700 1 -- 172.21.15.123:0/1961839249 shutdown_connections 2024-01-13T11:58:41.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:41.049+0000 7f9c59a49700 1 --2- 172.21.15.123:0/1961839249 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f9c3c02ff60 0x7f9c3c032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:41.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:41.049+0000 7f9c59a49700 1 --2- 172.21.15.123:0/1961839249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c540f5410 0x7f9c54106700 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:41.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:41.049+0000 7f9c59a49700 1 -- 172.21.15.123:0/1961839249 >> 172.21.15.123:0/1961839249 conn(0x7f9c540f0b70 msgr2=0x7f9c540f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:41.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:41.049+0000 7f9c59a49700 1 -- 172.21.15.123:0/1961839249 shutdown_connections 2024-01-13T11:58:41.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:41.049+0000 7f9c59a49700 1 -- 172.21.15.123:0/1961839249 wait complete. 2024-01-13T11:58:41.038 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:58:41.635 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:41 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:39.587082+0000 mgr.smithi059.hmyldj (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:41.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:41 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:58:41.048954+0000 mon.smithi059 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.123:0/1961839249' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:58:42.430 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:58:42.430 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3276948-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:58:43.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.607+0000 7f07d3c0b700 1 -- 172.21.15.123:0/641319248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07cc0f5410 msgr2=0x7f07cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:43.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.607+0000 7f07d3c0b700 1 --2- 172.21.15.123:0/641319248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07cc0f5410 0x7f07cc0f5830 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f07bc009fb0 tx=0x7f07bc009b30).stop 2024-01-13T11:58:43.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.607+0000 7f07d3c0b700 1 -- 172.21.15.123:0/641319248 shutdown_connections 2024-01-13T11:58:43.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.607+0000 7f07d3c0b700 1 --2- 172.21.15.123:0/641319248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07cc0f5410 0x7f07cc0f5830 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:43.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.607+0000 7f07d3c0b700 1 -- 172.21.15.123:0/641319248 >> 172.21.15.123:0/641319248 conn(0x7f07cc0f0b70 msgr2=0x7f07cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:43.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.607+0000 7f07d3c0b700 1 -- 172.21.15.123:0/641319248 shutdown_connections 2024-01-13T11:58:43.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.607+0000 7f07d3c0b700 1 -- 172.21.15.123:0/641319248 wait complete. 2024-01-13T11:58:43.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.608+0000 7f07d3c0b700 1 Processor -- start 2024-01-13T11:58:43.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.608+0000 7f07d3c0b700 1 -- start start 2024-01-13T11:58:43.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.608+0000 7f07d3c0b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07cc0f5410 0x7f07cc0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:43.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.608+0000 7f07d3c0b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc0fb820 con 0x7f07cc0f5410 2024-01-13T11:58:43.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.608+0000 7f07d19a7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07cc0f5410 0x7f07cc0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:43.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.608+0000 7f07d19a7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07cc0f5410 0x7f07cc0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.123:54730/0 (socket says 172.21.15.123:54730) 2024-01-13T11:58:43.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.608+0000 7f07d19a7700 1 -- 172.21.15.123:0/1558384146 learned_addr learned my addr 172.21.15.123:0/1558384146 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-13T11:58:43.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.608+0000 7f07d19a7700 1 -- 172.21.15.123:0/1558384146 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07bc005040 con 0x7f07cc0f5410 2024-01-13T11:58:43.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.609+0000 7f07d19a7700 1 --2- 172.21.15.123:0/1558384146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07cc0f5410 0x7f07cc0fd3a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f07bc013510 tx=0x7f07bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:43.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.609+0000 7f07c27fc700 1 -- 172.21.15.123:0/1558384146 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07bc009320 con 0x7f07cc0f5410 2024-01-13T11:58:43.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.609+0000 7f07d3c0b700 1 -- 172.21.15.123:0/1558384146 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07cc0fb9c0 con 0x7f07cc0f5410 2024-01-13T11:58:43.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.609+0000 7f07d3c0b700 1 -- 172.21.15.123:0/1558384146 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07cc0fbe50 con 0x7f07cc0f5410 2024-01-13T11:58:43.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.609+0000 7f07c27fc700 1 -- 172.21.15.123:0/1558384146 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07bc013f00 con 0x7f07cc0f5410 2024-01-13T11:58:43.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.609+0000 7f07c27fc700 1 -- 172.21.15.123:0/1558384146 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07bc014d60 con 0x7f07cc0f5410 2024-01-13T11:58:43.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.610+0000 7f07c27fc700 1 -- 172.21.15.123:0/1558384146 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07bc0095f0 con 0x7f07cc0f5410 2024-01-13T11:58:43.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.610+0000 7f07d3c0b700 1 -- 172.21.15.123:0/1558384146 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b4004fa0 con 0x7f07cc0f5410 2024-01-13T11:58:43.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.610+0000 7f07c27fc700 1 --2- 172.21.15.123:0/1558384146 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f07b802ff60 0x7f07b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-13T11:58:43.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.610+0000 7f07c27fc700 1 -- 172.21.15.123:0/1558384146 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07bc040f50 con 0x7f07cc0f5410 2024-01-13T11:58:43.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.610+0000 7f07d11a6700 1 --2- 172.21.15.123:0/1558384146 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f07b802ff60 0x7f07b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-13T11:58:43.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.611+0000 7f07d11a6700 1 --2- 172.21.15.123:0/1558384146 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f07b802ff60 0x7f07b8032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f07c80096a0 tx=0x7f07c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-13T11:58:43.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.615+0000 7f07c27fc700 1 -- 172.21.15.123:0/1558384146 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07bc06d2b0 con 0x7f07cc0f5410 2024-01-13T11:58:43.635 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:43 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: cluster 2024-01-13T11:58:41.587424+0000 mgr.smithi059.hmyldj (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:43.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.824+0000 7f07d3c0b700 1 -- 172.21.15.123:0/1558384146 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07b4005670 con 0x7f07cc0f5410 2024-01-13T11:58:43.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.825+0000 7f07c27fc700 1 -- 172.21.15.123:0/1558384146 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f07bc014420 con 0x7f07cc0f5410 2024-01-13T11:58:43.812 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-13T11:58:43.812 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"a3276948-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:45.820605Z","created":"2024-01-13T11:48:45.820605Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:58:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.826+0000 7f07d3c0b700 1 -- 172.21.15.123:0/1558384146 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f07b802ff60 msgr2=0x7f07b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.826+0000 7f07d3c0b700 1 --2- 172.21.15.123:0/1558384146 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f07b802ff60 0x7f07b8032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f07c80096a0 tx=0x7f07c8006b40).stop 2024-01-13T11:58:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.826+0000 7f07d3c0b700 1 -- 172.21.15.123:0/1558384146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07cc0f5410 msgr2=0x7f07cc0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-13T11:58:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.826+0000 7f07d3c0b700 1 --2- 172.21.15.123:0/1558384146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07cc0f5410 0x7f07cc0fd3a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f07bc013510 tx=0x7f07bc00b040).stop 2024-01-13T11:58:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.826+0000 7f07d3c0b700 1 -- 172.21.15.123:0/1558384146 shutdown_connections 2024-01-13T11:58:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.827+0000 7f07d3c0b700 1 --2- 172.21.15.123:0/1558384146 >> [v2:172.21.15.59:6800/32893793,v1:172.21.15.59:6801/32893793] conn(0x7f07b802ff60 0x7f07b8032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.827+0000 7f07d3c0b700 1 --2- 172.21.15.123:0/1558384146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07cc0f5410 0x7f07cc0fd3a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-13T11:58:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.827+0000 7f07d3c0b700 1 -- 172.21.15.123:0/1558384146 >> 172.21.15.123:0/1558384146 conn(0x7f07cc0f0b70 msgr2=0x7f07cc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-13T11:58:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.827+0000 7f07d3c0b700 1 -- 172.21.15.123:0/1558384146 shutdown_connections 2024-01-13T11:58:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-13T11:58:43.827+0000 7f07d3c0b700 1 -- 172.21.15.123:0/1558384146 wait complete. 2024-01-13T11:58:43.815 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-01-13T11:58:44.199 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-01-13T11:58:44.201 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-13T11:58:44.201 DEBUG:teuthology.orchestra.run.smithi059:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-13T11:58:44.226 DEBUG:teuthology.orchestra.run.smithi123:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-13T11:58:44.245 INFO:tasks.cephadm:Stopping all daemons... 2024-01-13T11:58:44.245 INFO:tasks.cephadm.mon.smithi059:Stopping mon.smithi059... 2024-01-13T11:58:44.245 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi059 2024-01-13T11:58:44.616 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:44 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: audit 2024-01-13T11:58:43.826111+0000 mon.smithi059 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.123:0/1558384146' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:58:44.616 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:44 smithi059 systemd[1]: Stopping Ceph mon.smithi059 for a3276948-b209-11ee-95ac-87774f69a715... 2024-01-13T11:58:44.616 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:44 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:58:44.394+0000 7f09f65ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi059 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-13T11:58:44.617 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:44 smithi059 ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059[38760]: debug 2024-01-13T11:58:44.394+0000 7f09f65ed700 -1 mon.smithi059@0(leader) e1 *** Got Signal Terminated *** 2024-01-13T11:58:44.885 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 13 11:58:44 smithi059 podman[44211]: ceph-a3276948-b209-11ee-95ac-87774f69a715-mon.smithi059 2024-01-13T11:58:44.895 DEBUG:teuthology.orchestra.run.smithi059:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi059.service' 2024-01-13T11:58:44.964 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-13T11:58:44.964 INFO:tasks.cephadm.mon.smithi059:Stopped mon.smithi059 2024-01-13T11:58:44.964 INFO:tasks.cephadm.mon.smithi123:Stopping mon.smithi123... 2024-01-13T11:58:44.964 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi123 2024-01-13T11:58:44.999 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi123.service: Unit ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi123.service not loaded. 2024-01-13T11:58:45.000 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-01-13T11:58:45.001 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi123" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi123 with status 5: 'sudo systemctl stop ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi123' 2024-01-13T11:58:45.002 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-13T11:58:45.003 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516050/remote/ubuntu@smithi059.front.sepia.ceph.com/crash 2024-01-13T11:58:45.004 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/crash -- . 2024-01-13T11:58:45.055 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516050/remote/ubuntu@smithi123.front.sepia.ceph.com/crash 2024-01-13T11:58:45.057 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/crash -- . 2024-01-13T11:58:45.097 INFO:teuthology.orchestra.run.smithi123.stderr:tar: /var/lib/ceph/a3276948-b209-11ee-95ac-87774f69a715/crash: Cannot open: No such file or directory 2024-01-13T11:58:45.097 INFO:teuthology.orchestra.run.smithi123.stderr:tar: Error is not recoverable: exiting now 2024-01-13T11:58:45.098 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-13T11:58:45.098 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-13T11:58:45.130 INFO:teuthology.orchestra.run.smithi059.stderr:grep: /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715/ceph.log: No such file or directory 2024-01-13T11:58:45.132 INFO:tasks.cephadm:Compressing logs... 2024-01-13T11:58:45.132 DEBUG:teuthology.orchestra.run.smithi059:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-13T11:58:45.174 DEBUG:teuthology.orchestra.run.smithi123:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-13T11:58:45.201 INFO:teuthology.orchestra.run.smithi059.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-13T11:58:45.201 INFO:teuthology.orchestra.run.smithi059.stderr:: No such file or directory 2024-01-13T11:58:45.201 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715/ceph-mon.smithi059.log 2024-01-13T11:58:45.201 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715/ceph-volume.log 2024-01-13T11:58:45.201 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715/ceph-mgr.smithi059.hmyldj.log 2024-01-13T11:58:45.202 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715/ceph-mon.smithi059.log: /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715/ceph-volume.log: /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715/ceph-mgr.smithi059.hmyldj.log: 88.3% -- replaced with /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715/ceph-volume.log.gz 2024-01-13T11:58:45.202 INFO:teuthology.orchestra.run.smithi123.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-13T11:58:45.202 INFO:teuthology.orchestra.run.smithi123.stderr:: No such file or directory 2024-01-13T11:58:45.204 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/cephadm.log: 96.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-13T11:58:45.205 INFO:teuthology.orchestra.run.smithi059.stderr: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-13T11:58:45.206 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-01-13T11:58:45.206 INFO:teuthology.orchestra.run.smithi123.stderr:real 0m0.014s 2024-01-13T11:58:45.206 INFO:teuthology.orchestra.run.smithi123.stderr:user 0m0.011s 2024-01-13T11:58:45.206 INFO:teuthology.orchestra.run.smithi123.stderr:sys 0m0.011s 2024-01-13T11:58:45.281 INFO:teuthology.orchestra.run.smithi059.stderr: 92.7% -- replaced with /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715/ceph-mgr.smithi059.hmyldj.log.gz 2024-01-13T11:58:45.352 INFO:teuthology.orchestra.run.smithi059.stderr: 90.5% -- replaced with /var/log/ceph/a3276948-b209-11ee-95ac-87774f69a715/ceph-mon.smithi059.log.gz 2024-01-13T11:58:45.354 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-13T11:58:45.354 INFO:teuthology.orchestra.run.smithi059.stderr:real 0m0.164s 2024-01-13T11:58:45.354 INFO:teuthology.orchestra.run.smithi059.stderr:user 0m0.238s 2024-01-13T11:58:45.354 INFO:teuthology.orchestra.run.smithi059.stderr:sys 0m0.021s 2024-01-13T11:58:45.354 INFO:tasks.cephadm:Archiving logs... 2024-01-13T11:58:45.354 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516050/remote/ubuntu@smithi059.front.sepia.ceph.com/log 2024-01-13T11:58:45.355 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-13T11:58:45.479 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516050/remote/ubuntu@smithi123.front.sepia.ceph.com/log 2024-01-13T11:58:45.480 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-13T11:58:45.509 INFO:tasks.cephadm:Removing cluster... 2024-01-13T11:58:45.509 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a3276948-b209-11ee-95ac-87774f69a715 --force 2024-01-13T11:59:43.409 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a3276948-b209-11ee-95ac-87774f69a715 --force 2024-01-13T11:59:43.724 INFO:tasks.cephadm:Removing cephadm ... 2024-01-13T11:59:43.725 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-13T11:59:43.745 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-13T11:59:43.763 INFO:tasks.cephadm:Teardown complete 2024-01-13T11:59:43.763 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi123 with status 5: 'sudo systemctl stop ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi123' 2024-01-13T11:59:43.991 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f5fc72132aba48b987d032dc17d4ac96 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi123 with status 5: 'sudo systemctl stop ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi123' 2024-01-13T11:59:43.996 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-13T11:59:44.011 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-13T11:59:44.018 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-13T11:59:44.018 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-13T11:59:44.034 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-13T11:59:44.115 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-13T11:59:44.130 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-13T11:59:44.130 DEBUG:teuthology.orchestra.run.smithi059:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T11:59:44.133 DEBUG:teuthology.orchestra.run.smithi123:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T11:59:44.150 INFO:teuthology.orchestra.run.smithi123.stderr:bash: ntpq: command not found 2024-01-13T11:59:44.151 INFO:teuthology.orchestra.run.smithi059.stderr:bash: ntpq: command not found 2024-01-13T11:59:44.154 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-13T11:59:44.154 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-01-13T11:59:44.154 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 10 -673us[ -673us] +/- 83ms 2024-01-13T11:59:44.154 INFO:teuthology.orchestra.run.smithi123.stdout:^* hv02.front.sepia.ceph.com 3 6 377 10 -52us[ -52us] +/- 58ms 2024-01-13T11:59:44.154 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 +685us[ +685us] +/- 76ms 2024-01-13T11:59:44.155 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T11:59:44.156 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-13T11:59:44.156 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2024-01-13T11:59:44.156 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 -673us[ -656us] +/- 83ms 2024-01-13T11:59:44.156 INFO:teuthology.orchestra.run.smithi059.stdout:^* hv02.front.sepia.ceph.com 3 6 377 10 -53us[ -37us] +/- 58ms 2024-01-13T11:59:44.156 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 +726us[ +743us] +/- 76ms 2024-01-13T11:59:44.156 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T11:59:44.157 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-13T11:59:44.167 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-13T11:59:44.168 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-13T11:59:44.177 DEBUG:teuthology.orchestra.run.smithi059:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-13T11:59:44.267 DEBUG:teuthology.orchestra.run.smithi123:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-13T11:59:44.330 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-13T11:59:44.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:59:44.363 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-13T11:59:44.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:59:44.393 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-13T11:59:44.404 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-13T11:59:44.415 INFO:teuthology.task.internal:Duration was 1284.212385 seconds 2024-01-13T11:59:44.415 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-13T11:59:44.425 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-13T11:59:44.425 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-13T11:59:44.428 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-13T11:59:44.476 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-13T11:59:44.478 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-13T11:59:44.679 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-13T11:59:44.680 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi059.front.sepia.ceph.com 2024-01-13T11:59:44.680 DEBUG:teuthology.orchestra.run.smithi059:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-13T11:59:44.705 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi123.front.sepia.ceph.com 2024-01-13T11:59:44.705 DEBUG:teuthology.orchestra.run.smithi123:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-13T11:59:44.728 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-13T11:59:44.728 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-13T11:59:44.748 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-13T11:59:44.816 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-13T11:59:44.816 DEBUG:teuthology.orchestra.run.smithi059:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-13T11:59:44.818 DEBUG:teuthology.orchestra.run.smithi123:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-13T11:59:44.906 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-13T11:59:44.917 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-13T11:59:44.918 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-13T11:59:44.946 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-13T11:59:44.972 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-13T11:59:44.983 DEBUG:teuthology.orchestra.run.smithi059:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-13T11:59:44.989 DEBUG:teuthology.orchestra.run.smithi123:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-13T11:59:45.014 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = core 2024-01-13T11:59:45.039 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = core 2024-01-13T11:59:45.064 DEBUG:teuthology.orchestra.run.smithi059:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-13T11:59:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:59:45.096 DEBUG:teuthology.orchestra.run.smithi123:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-13T11:59:45.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:59:45.122 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-13T11:59:45.132 INFO:teuthology.task.internal:Transferring archived files... 2024-01-13T11:59:45.133 DEBUG:teuthology.misc:Transferring archived files from smithi059:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516050/remote/smithi059 2024-01-13T11:59:45.134 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-13T11:59:45.178 DEBUG:teuthology.misc:Transferring archived files from smithi123:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516050/remote/smithi123 2024-01-13T11:59:45.179 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-13T11:59:45.217 INFO:teuthology.task.internal:Removing archive directory... 2024-01-13T11:59:45.218 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-13T11:59:45.220 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-13T11:59:45.269 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-13T11:59:45.281 INFO:teuthology.task.internal:Not uploading archives. 2024-01-13T11:59:45.282 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-13T11:59:45.291 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-13T11:59:45.291 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-13T11:59:45.294 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-13T11:59:45.309 INFO:teuthology.orchestra.run.smithi059.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 13 11:59 /home/ubuntu/cephtest 2024-01-13T11:59:45.327 INFO:teuthology.orchestra.run.smithi123.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 13 11:59 /home/ubuntu/cephtest 2024-01-13T11:59:45.328 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-13T11:59:45.339 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-13T11:59:45.360 INFO:teuthology.nuke:Checking targets against current locks 2024-01-13T11:59:45.388 DEBUG:teuthology.nuke:shortname: smithi059 2024-01-13T11:59:45.389 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-13T11:59:45.413 DEBUG:teuthology.nuke:shortname: smithi123 2024-01-13T11:59:45.413 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-13T11:59:45.417 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516050', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 11:29:34.001224', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLe/qubIVW8mPTHtv8LfzeNUpJlwG55ekiV/OKHpQSrMAuXJfj3B67+ixTM7TOPMnqW5mNj0YIBWLlH9Pu46Ds0='} 2024-01-13T11:59:45.440 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516050', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 11:29:34.002840', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIIIhmroZg0ERs05ifmQozDoK5B7BmGFr/SA5Tn+H2Xs2WcmrNNhpZY2zOiY28R1ZginFOa+cPP3jrsRA6fVz28='} 2024-01-13T11:59:45.470 INFO:teuthology.orchestra.console.smithi059:Power off 2024-01-13T11:59:45.471 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-13T11:59:45.494 DEBUG:teuthology.orchestra.console.smithi059:power off output: Chassis Power Control: Down/Off 2024-01-13T11:59:45.494 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T11:59:45.510 INFO:teuthology.orchestra.console.smithi123:Power off 2024-01-13T11:59:45.511 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-13T11:59:45.527 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2024-01-13T11:59:45.534 DEBUG:teuthology.orchestra.console.smithi123:power off output: Chassis Power Control: Down/Off 2024-01-13T11:59:45.534 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T11:59:45.557 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-01-13T11:59:49.529 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T11:59:49.558 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T11:59:49.641 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2024-01-13T11:59:49.668 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-01-13T11:59:53.646 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T11:59:53.669 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T11:59:53.756 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-01-13T11:59:53.779 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is off 2024-01-13T11:59:53.857 INFO:teuthology.orchestra.console.smithi059:Power off completed 2024-01-13T11:59:53.880 INFO:teuthology.orchestra.console.smithi123:Power off completed 2024-01-13T11:59:53.985 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1284.2123854160309 failure_reason: 'Command failed on smithi123 with status 5: ''sudo systemctl stop ceph-a3276948-b209-11ee-95ac-87774f69a715@mon.smithi123''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f5fc72132aba48b987d032dc17d4ac96 status: fail success: false 2024-01-13T11:59:53.986 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-13T11:59:54.072 INFO:teuthology.run:FAIL